Committee of Visitors Report

for the

NSF/GEO/ATM/

UCAR and Lower Atmospheric Facilities Oversight Section

August 18-19, 2003

# OVERVIEW, 

COV REVIEW PROCESS

The Committee of Visitors (COV) review of the NSF/GEO/ATM's UCAR and
Lower Atmosphere Facilities Oversight Section (ULAFOS) was conducted at
the National Science Foundation in Arlington, Virginia, August 18-19,
2003. The charge to the Committee is contained in the letter to
Committee members from Assistant Director for Geosciences Dr. Margaret
Leinen dated July 30, 2003 Members of the COV were:

Elbert (Joe) Friday, National Research Council, (Chair)

Estelle Condon, NASA Ames Research Center

Fredrick H. M. Semazzi, North Carolina State University

Susan L. Ustin, University of California, Davis

R Lynn Rose, L3/Aeromet, Inc

George Seielstad, University of North Dakota

This is the third COV review of ULAFOS, the last one having been
conducted in August, 2000 (chair: Inez Fung).

Materials for the review consisted of: (i) a notebook of background
information (including a CD-ROM) provided by the ULAFOS office; (ii)
presentations by the ULAFOS staff; and (iii) a meeting with Jarvis
Moyers, Director of ATM. Also available for inspection by the COV were
reviews administered by ULAFOS in the past several years for the
cooperative agreements.

**[MAJOR FINDINGS AND ISSUES DESERVING IMMEDIATE
ATTENTION]{.underline}**

The COV wishes to expressly draw the following findings and issues to
the NSF management's attention. Several other conclusions and
recommendations are contained throughout the document for your
consideration.

**[MAJOR FINDINGS]{.underline}**

The ULAFOS staff is doing an outstanding job of managing extremely
important national resources.

The programs ULAFOS manage contribute significantly to the nation's
scientific strength and to the strategic goals of the NSF.

The management processes used by ULAFOS meet the highest standards set
by NSF.

**[ISSUES DESERVING IMMEDIATE ATTENTION]{.underline}**

The cooperative agreement for UCAR's management of NCAR needs to be
awarded by 1 October 2003. Based on present progress, the COV is
concerned that this will not be completed in time and continuity of
operations will be jeopardized.

The ULAFOS staff is very efficient and productive, but the present IPA
needs to be made into a permanent position. A succession plan needs to
be developed. Sufficient resources must be provided (in number, quality,
areas of expertise) to support the planned re-competition of the NCAR
management cooperative agreement. The amount of work required for an
open competition for this cooperative agreement cannot be
over-emphasized.

The COV strongly supports the development of an integrated facilities
plan to incorporate interagency assets. Even though the other agency's
assets may not be managed by ULAFOS, their presence can be factored into
the overall capability to support the atmospheric science research in
this country. Such a plan should bring more assets to bear for the
research community.

The COV endorses the independent review of the mechanisms and modes of
support for facilities, research and education in the atmospheric and
related sciences. This review is not a simple task. The COV recommends
that it be conducted by a high level independent senior body, and urges
that the issues identified in section C-1 of this report be included in
the review.

# Unique responsibilities and functions of the ULAFOS section

ULAFOS funds and administers five major programs including four
Cooperative Agreements (CA's):

UCAR, including NCAR and the UCAR Office of Programs,

Colorado State University for the CHILL Dual Doppler Radar,

South Dakota School of Mines and Technology for the T-28 Storm
Penetration Aircraft,

University of Wyoming for the King Air research aircraft, and

A deployment pool of approximately \$3 million to support field
experiments.

The bulk of the UCAR activities under ULAFOS responsibilities are at
NCAR and some such as UNIDATA and the Joint Office for Science Support
(JOSS) are under the UCAR Office of Programs (UOP). Facilities for
observing the lower atmosphere are provided by three universities in
support of peer-reviewed research. These facilities are available free
of charge to principal investigators of NSF-funded research. Placing
these facilities at universities enables a closer connection between
facilities and education and training.

Thus, ULAFOS by and large does not oversee a traditional competitive
grants program. Since its inception, NCAR has operated as a research and
facility center whose infrastructure (scientific computing, atmospheric
models, observational facilities) has been utilized by the entire
atmospheric and related sciences community. In the early 1970's, NCAR
was designated as a Federally Funded Research and Development Center
(FFRDC), the only FFRDC within GEO. UCAR is a consortium of universities
that has been selected by NSF to manage NCAR.

ULAFOS Section Head Dr. Cliff Jacobs and his very small staff of four
(James Huning, Bernard Grant, Peter Milne and Carolyn Walton) are to be
commended for their excellent job of administration of their programs.
The very positive professional relationship between the ULAFOS office
and UCAR/NCAR senior staff clearly has advanced the science and the
services provided to the community.

The COV finds that renewal of the Cooperative Agreement was carried out
with integrity and thoroughness in accordance with NSF merit review
procedures, in a timely manner. The recommendations for the award of the
UCAR five-year cooperative agreement were submitted to and approved by
the NSB. **The COV notes that the cooperative agreement has still not
been awarded at the time of our meeting, and is concerned with the
ability of the NSF to complete the award by October 1, 2003.**

The COV finds the quality of the results of the NSF funded, ULAFOS
managed programs is excellent. The COV further finds that ULAFOS
performance has met the GPRA Goals very successfully and has made
substantial contributions towards GPRA Outcomes. The recommendations we
provide in this report are intended to improve the overall performance
of the ULAFOS and its programs.

The COV understands and views positively the decision by the NSB not to
open the proposal renewal process for the Cooperative Agreement for
2003-2008, but understands that the intent is to open the proposal
process to competition at the next renewal opportunity. The COV urges
NSF to provide sufficient resources to ULAFOS to properly prepare for an
open competition that will replace an arrangement that has been in place
for over 45 years with very positive results.

For the reasons stated above, the review of ULAFOS cannot and should not
be conducted in the same way as that of a competitive grants program.
The COV unanimously believes that the highly structured review process
specified by the report template for NSF COVs does not apply well here.
Accordingly, while the ULAFOS COV followed the specified template as
best it could, it departed from the template where that seemed
appropriate. Further, we note that the previous COV made the same
observation and recommended that an appropriate review process be
established for ULAFOS and other facility-intensive NSF units. Although
the Template used this year differs from that in the previous review,
the conclusion still holds.

##  Integrity and Efficiency of the Program's Processes & Management

The combination of anonymous reviews and panel site visits is considered
optimal. ULAFOS has made considerable progress in shortening the review
process, especially for NCAR/UCAR. The time and effort required by the
review is extensive, but the size and complexity of the organization
demands it. The review criteria in the previous review of UCAR/NCAR were
well stated and well addressed during the review process. The selection
of the reviewers and panel members was balanced in every aspect. The COV
concluded that the panel reviews were particularly productive.

ULAFOS oversees four awards, in the form of Cooperative Agreements and
the Deployment Pool. While this number may appear small, these awards
require intense, interactive, on-going effort on the part of the ULAFOS
staff. While the interaction is particularly intense during review and
award time, it is on-going for the duration of the Agreement. The COV is
impressed with the volume and quality of work accomplished by the very
small staff of ULAFOS. The COV observes the staff to be well qualified
and extremely productive, but is concerned that one key staff position
is filled, not by a permanent appointment, but rather an IPA. **Even
though the COV noted that the present IPA is extremely productive, in a
small staff of this nature, dealing with such a vital activity to the
atmospheric community, the COV recommends that this position be
converted to a full time permanent appointment to promote continuity.
Similarly, the COV recommends the development of a succession plan for
the section.**

## Outputs and Outcomes of NSF Investments

The outputs and outcomes of NSF investments are fully documented in the
various reports and materials contained in the UCAR/NCAR folder examined
by the committee. We begin by quoting from the review panel report to
NSF on the UCAR proposal for NCAR management, December 19 2002 that
exemplifies the essence of the final review of UCAR/NCAR:

> Previous reviews, including the most recent NSF and SPEC reviews, have
> noted the excellence of NCAR research. This panel concurs in that
> assessment and is impressed with the quality and appropriateness of
> the proposed research agenda. This agenda clearly shows intimate
> understanding of the needs and opportunities in the field of
> atmospheric sciences and reflects priorities that are consistent with
> the NCAR/UCAR mission and vision, with the NSF geosciences documents,
> and with the NRC Atmospheric Sciences document "Entering the
> Twenty-first Century".

## Response to previous COV report

The previous COV made several recommendations in its report of August
2000 ULAFOS provided an initial response to those recommendations in
October 2000. During the previous three years, progress has been made
towards improving elements of the programs identified by the previous
COV to which ULAFOS provides oversight and the operations of the
Section.

The review process used in 2001-2002 was streamlined from the previous
review and concluded the science/facilities and management reviews in
three months. As stated in the October 2000 response to the COV report,
ULAFOS continues to believe that a three phased review is necessary for
NSF requirements for a credible, independent, and balanced review of
NCAR and the UCAR management thereof. A two phase review such as
suggested by the previous COV would not allow management issues
identified by reviewers to be addressed before requesting approval to
make an award. For example, the management review panel of 2001 raised
two concerns which were addressed by UCAR in 2002, viz., a lack of a
strategic plan for laboratory and office space and the need for an
independent management of the HIAPER project. In May 2003, ULAFOS was
able to report to the National Science Board significant progress in
addressing these concerns. **The present COV concurs that the present
three stage review has merit, but recommends continuing efforts to
assure a timely review.**

The COV notes two other recent factors may modify the procedures that
ULAFOS uses for the UCAR review process. The first of these is the
expressed intention, according to the policy of the National Science
Board ([NSB 97-224]{.underline}), to compete the management of NCAR at
the end of the next cooperative agreement (in 2008). The staffing needs
associated with the open competition have already been mentioned above.
The second of these concerns a recent NSB resolution that directs the
Division of Atmospheric Sciences to conduct a review of the mechanisms
and modes of support for facilities, research and education in the
atmospheric and related sciences. The COV strongly endorses such a
detailed review, suggests that it be conducted by a high level
independent senior body, and urges that the issues identified in section
C-3 of this report be included in the review.

##### FY 2003 REPORT TEMPLATE 

**FOR THE ULAFOS COV**

+-----------------------------------------------------------------------+
| ##### Date of COV Aug 18-19, 2003                                     |
+-----------------------------------------------------------------------+
| **Program/Cluster: UCAR & Lower Atmospheric Facilities**              |
+-----------------------------------------------------------------------+
| **Division: Atmospheric Sciences**                                    |
+-----------------------------------------------------------------------+
| **Directorate: Geosciences**                                          |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^1]**: Awards: 3 Declinations:  |
| 0 Other:\>100's**                                                     |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^2]**: Awards: 5+ Declinations: 0      |
| Other: \>100's**                                                      |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected:**                   |
|                                                                       |
| **Because of the specific functions of the ULAFOS section (oversight  |
| of relatively few, yet very large and or complex, awards in the form  |
| of cooperative agreements) recent representative processes were       |
| selected for review. These centered on a) renewal of the UCAR         |
| cooperative agreement for the operation of NCAR, b) renewal of the    |
| Unidata cooperative agreement, c) the OFAP facilities review process  |
| for deployment pool funds.**                                          |
+-----------------------------------------------------------------------+

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. Provide comments in the space below the
question. Discuss areas of concern in the space provided.

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | **YES**  |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments: **The review structure established is an        |          |
| appropriate mix of site visits, reviews, etc., for the    |          |
| types of contracts monitored by ULAFOS.**                 |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | **YES**  |
|                                                           |          |
| Comments: **The process is efficient and effective. The   |          |
| size of the programs supported by ULAFOS is orders of     |          |
| magnitude greater than the typical NSF grant or contract. |          |
| The process takes longer, but is appropriate to the size  |          |
| of the programs**                                         |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | **YES**  |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments**: The combination of mail reviews and panel     |          |
| reviews is consistent with NSF priorities and criteria    |          |
| and those set by** **the NSB. Management of UCAR and NCAR |          |
| differs from standard NSF competitive reviews. This COV   |          |
| supports prior recommendations for modifying the NSF      |          |
| template review to better accommodate the advice needed   |          |
| by NSF program managers who have major responsibility for |          |
| oversight and administration of** **cooperative           |          |
| agreements and contracts.**                               |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | **YES**  |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments: **Mail and panel reviews provide sufficiently   |          |
| detailed information for UCAR/NCAR to understand NSF      |          |
| recommendations and take appropriate management           |          |
| actions.**                                                |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | **YES**  |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
|                                                           |          |
| Comments: **The previous ULAFOS COV and 2002 Panel        |          |
| on-site review of UCAR and NCAR proposals provided        |          |
| sufficiently detailed panel summaries for UCAR/NCAR to    |          |
| understand NSF recommendations and take appropriate       |          |
| management actions.**                                     |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | **YES**  |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments: **There is ample documentation from the program |          |
| office to justify the CA recommendation from the review   |          |
| materials. The renewal of the CA was conducted with       |          |
| integrity and thoroughness in accordance with NSF merit   |          |
| review procedures and** **the quality of the reviews was  |          |
| excellent.**                                              |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | **YES**  |
|                                                           |          |
| Comments: **The separate reviews of the NCAR divisions,   |          |
| UCAR management and the NCAR proposal extended the        |          |
| duration of the review process. The COV believes that     |          |
| ULAFOS completed their portion of the activity in a       |          |
| timely manner, but is very concerned that the contract    |          |
| award has not yet been made!**                            |          |
+-----------------------------------------------------------+----------+
| > Discuss issues identified by the COV concerning the     |          |
| > quality and effectiveness of the program's use of merit |          |
| > review procedures:                                      |          |
| >                                                         |          |
| > **The COV is concerned with the time taken to award the |          |
| > present Cooperative Agreement to UCAR for the           |          |
| > management of NCAR. We are therefore especially         |          |
| > concerned about the ability of NSF to effect an open    |          |
| > competition for the 2008 competition for NCAR           |          |
| > management. THE COV RECOMMENDS THAT SUFFICIENT          |          |
| > RESOURCES (IN NUMBERS, QUALITY, AND APPROPRIATE AREAS   |          |
| > OF EXPRETISE) BE PROVIDED TO ULAFOS TO BEGIN THE        |          |
| > PROCESS WITH SUFFICIENT PLANNING AND PREPARATION** **TO |          |
| > ASSURE A** **QUALITY COMPETITION. THIS NATIONAL ASSET   |          |
| > DEMANDS NOTHING LESS.**                                 |          |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. Provide comments in the space below the question.
Discuss issues or concerns in the space provided.

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | **YES**   |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: **The mail reviewers and panel for the         |           |
| UCAR/NCAR proposals addressed both intellectual merit    |           |
| and broader impacts review criteria.**                   |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | **YES**   |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: **The panel summary directly addressed how     |           |
| UCAR/NCAR contributed to both merit review criteria.**   |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   | **YES**   |
| the proposal contributes to both merit review criteria?  |           |
|                                                          |           |
| Comments: **The review analyses meets NSF expectations   |           |
| regarding this item.**                                   |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| **This committee supports the concerns that were         |           |
| identified by the previous COV in its review. The        |           |
| previous COV identified several issues with the NSF      |           |
| merit review system for UCAR/NCAR. They noted that the   |           |
| three-year periodicity of the COV review cycle does not  |           |
| match the five year Cooperative Agreement cycle and that |           |
| this creates a large review burden. They also note that  |           |
| the "standard" review questions (e.g., A2, A3, and A4)   |           |
| in the COV Report Template do not fit the needs of       |           |
| ULAFOS for its review of the UCAR/NCAR program.**        |           |
|                                                          |           |
| **The COV notes that the ULAFOS has gained some          |           |
| flexibility in the NSF guidelines to allow the COV       |           |
| evaluating ULAFOS to apply only the portions of the      |           |
| evaluation template appropriate to the activities of the |           |
| Section.**                                               |           |
+----------------------------------------------------------+-----------+

A.3 Questions concerning the selection of reviewers. Provide comments in
the space below the question. Discuss areas of concern in the space
provided.

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | YES        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: **A large number of reviewers provided mail   |            |
| and panel reviews for NCAR's proposal renewal and       |            |
| divisional reviews. Reviewers came from a wide range of |            |
| government labs, industry and universities.**           |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | YES        |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: **The expertise of reviewers spanned a wide   |            |
| range of appropriate scientific disciplines. The        |            |
| previous COV noted that UCAR/NCAR is a complex          |            |
| organization conducting a wide range of activities.     |            |
| Therefore, some knowledge and appreciation of NCAR's    |            |
| activities prior to the review is critical. The COV     |            |
| noted that an overly strict application of the          |            |
| procedure for qualifying reviewers could eliminate      |            |
| qualified individuals from the reviewer pool.**         |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | YES        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: **The NSF review included reviewers from a    |            |
| wide range of institutions, geographic locations, and   |            |
| underrepresented groups.**                              |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | YES        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: **The program eliminated anyone with any      |            |
| potential conflict of interest from the review.**       |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| Because of the large number of reviewers needed to      |            |
| complete all of the proposal renewal and divisional     |            |
| reviews and the expertise required to span the          |            |
| UCAR/NCAR program, the 2000 COV recommended that ULAFOS |            |
| adopt the conflict of interest methodology used by the  |            |
| National Academy of Sciences rather than simply         |            |
| disqualifying potential reviewers. This COV understands |            |
| this recommendation conflicts with overall NSF policy,  |            |
| and feels that the methods used by ULAFOS for this      |            |
| review cycle accomplished the goal of attaining the     |            |
| required number of reviewers without violating NSF      |            |
| procedures.                                             |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under review.
Provide comments in the space below the question. Discuss areas of
concern in the space provided.

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | **            |
| projects supported by the program.                   | APPROPRIATE** |
|                                                      |               |
| Comments: **The COV finds the awards made by ULAFOS  |               |
| over the past three years of extremely high quality, |               |
| as documented in the in the background material      |               |
| prepared for the COV, various reports, and materials |               |
| available from a broad range of sources. The overall |               |
| quality of the research and educational projects are |               |
| highly responsive to NSF mission. Through Scientific |               |
| Program Orders (SPO) funding mechanism ULAFOS        |               |
| supports a variety of high impact research and       |               |
| educational projects. These include, observational   |               |
| research platforms; collection, archiving and        |               |
| dissemination of research data; scientific           |               |
| visualization; High performance computing;           |               |
| development of community prediction models, COSMIC;  |               |
| a diverse array of activities in Global Change       |               |
| Research; and cross-fertilization research           |               |
| collaboration of atmospheric science with supporting |               |
| disciplines, such as statistics. The UCAR            |               |
| Educational Outreach Program SPO supports UCAR\'s    |               |
| Visiting Scientist Program; Postdoctoral Program in  |               |
| Ocean Modeling; Cooperative Program for Operational  |               |
| Meteorology Education and Training (COMET);          |               |
| Significant Opportunities in Atmospheric Research;   |               |
| NCAR Project to teach teachers; Virtual              |               |
| Exploratorium, and Science (SOARS) and other UCAR    |               |
| activities. HIAPER is a new initiative based on a    |               |
| high performance mid-size jet aircraft.**            |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | **            |
| scope of the projects?                               | APPROPRIATE** |
|                                                      |               |
| Comments: **Unique situation, however, the COV       |               |
| strongly urges the NSF, during the open              |               |
| recompetition for the management of NCAR, to seek a  |               |
| 5-year** **cooperative agreement** **with the option |               |
| for at least one 5-year extension. This will greatly |               |
| reduce the overhead costs of the procurements and    |               |
| will minimize disruption of the activities of NCAR   |               |
| and its manager.**                                   |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | APPROPRIATE** |
|                                                      |               |
| -   High Risk Proposals?                             |               |
|                                                      |               |
| Comments: **Through Scientific Program Orders (SPO)  |               |
| funding mechanism ULAFOS supports a variety of high  |               |
| impact research and educational projects.**          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | APPROPRIATE** |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
|                                                      |               |
| Comments: **NCAR/UCAR provides an effective          |               |
| environment for interdisciplinary research in        |               |
| atmospheric sciences.**                              |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | APPROPRIATE** |
|                                                      |               |
| -   Innovative Proposals?                            |               |
|                                                      |               |
| Comments: **Significant innovation in the NCAR       |               |
| vision for the future.**                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **na**        |
| balance of:                                          |               |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
|                                                      |               |
| Comments: **Applicable to typical NSF grants         |               |
| programs, not to ULAFOS.**                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | APPROPRIATE** |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| Comments: **The new director's opportunity fund at   |               |
| NCAR supports new investigators and investigations** |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **na**        |
| balance of:                                          |               |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| Comments: **Applicable to typical NSF grants         |               |
| programs, not to ULAFOS.**                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **na**        |
| balance of:                                          |               |
|                                                      |               |
| -   Institutional types?                             |               |
|                                                      |               |
| Comments: **Applicable to typical NSF grants         |               |
| programs, not to ULAFOS.**                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | APPROPRIATE** |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
|                                                      |               |
| Comments: **Excellent integrated programs.**         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance:                                             | APPROPRIATE** |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
|                                                      |               |
| Comments: **Encouraged by the director's opportunity |               |
| fund**                                               |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | **            |
| participation of underrepresented groups?            | APPROPRIATE** |
|                                                      |               |
| Comments: **The UCAR Educational Outreach Program    |               |
| SPO supports UCAR\'s Visiting Scientist Program;     |               |
| Postdoctoral Program in Ocean Modeling; Cooperative  |               |
| Program for Operational Meteorology Education and    |               |
| Training (COMET); Significant Opportunities in       |               |
| Atmospheric Research (SOARS); NCAR Project to teach  |               |
| teachers; Virtual Exploratorium, and other UCAR      |               |
| activities.**                                        |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | **            |
| agency mission, relevant fields and other customer   | APPROPRIATE** |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: **Quoting from the review panel report to  |               |
| NSF on the UCAR proposal for NCAR management,        |               |
| December 19 2002, that exemplifies the essence of    |               |
| the final review of UCAR/NCAR:**                     |               |
|                                                      |               |
| > **Previous reviews, including the most recent NSF  |               |
| > and SPEC reviews, have noted the excellence of     |               |
| > NCAR research. This panel concurs in that          |               |
| > assessment and is impressed with the quality and   |               |
| > appropriateness of the proposed research agenda.   |               |
| > This agenda clearly shows intimate understanding   |               |
| > of the needs and opportunities in the field of     |               |
| > atmospheric sciences and reflects priorities that  |               |
| > are consistent with the NCAR/UCAR mission and      |               |
| > vision, with the NSF geosciences documents, and    |               |
| > with the NRC Atmospheric Sciences document         |               |
| > "Entering the Twenty-first Century".**             |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
+------------------------------------------------------+---------------+

A.5 Management of the program under review. Please comment on:

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: ***[The COV finds the management of the programs reviewed   |
| is outstanding.]{.underline}***                                       |
|                                                                       |
| **ULAFOS oversees four awards, in the form of Cooperative Agreements  |
| and the Deployment Pool. While this number may appear small, these    |
| awards require intense, interactive, on-going effort on the part of   |
| the ULAFOS staff. While the interaction is particularly intense       |
| during review and award time, it is on-going for the duration of the  |
| Agreement. The COV is impressed with the volume and quality of work   |
| accomplished by the very small staff of ULAFOS. The COV observes the  |
| staff to be well qualified and extremely productive, but is concerned |
| that one key staff position is filled, not by a permanent position,   |
| but rather as IPA. In a small staff of this nature, dealing with such |
| a vital activity to the atmospheric community, the COV recommends     |
| that this position be converted to a full time position to promote    |
| continuity. Similarly, the COV recommends the development of a        |
| succession plan for the section.**                                    |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: **The NCAR strategic plan effectively addressed emerging    |
| science issues.**                                                     |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: **The NCAR strategic plan had broad review and buy-in by    |
| the** **wider** **community.**                                        |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The COV finds that renewal of the Cooperative Agreements was carried  |
| out with integrity and thoroughness in accordance with NSF merit      |
| review procedures, in a timely manner. The recommendations for the    |
| award of the UCAR five-year cooperative agreement were submitted to   |
| and approved by the NSB. **[The COV notes that the cooperative        |
| agreement has still not been awarded at the time of our meeting, and  |
| is concerned with the ability of the NSF to complete the award by     |
| October 1, 2003.]{.underline}**                                       |
|                                                                       |
| The COV understands and views positively the decision by the NSB not  |
| to open the proposal renewal process for the Cooperative Agreement    |
| for 2003-2008, but understands that the intent is to open the         |
| proposal process to competition at the next renewal opportunity. [The |
| COV urges NSF to provide sufficient resources to ULAFOS to properly   |
| prepare for an open competition for an arrangement that has been in   |
| place for over 45 years with very positive results. These resources   |
| must include expertise in]{.underline} [procedures that are dedicated |
| to this very complex procurement.]{.underline}                        |
+-----------------------------------------------------------------------+

**UNIDATA**

The COV also reviewed the UNIDATA activity managed by ULAFOS. Instead of
filling in the template as with the NCAR/UCAR review, the discussion is
presented in broad paragraph format corresponding to the subjects
covered by the template:

**[OVERALL FINDING]{.underline}**

The COV commends ULAFOS for their oversight of this vital national
resource. The broad community acceptance and the ubiquitous use of this
facility demonstrate the power of such a consortium of universities
managing the NCAR and UNIDATA activities.

**A1. Integrity and Efficiency of Merit Review Procedures used by
ULAFOS**

The COV examined a Case Study of treatment of the proposal "Unidata
2008: Shaping the future of data use in the geosciences." The integrity
and efficiency with which this proposal was reviewed met the highest
standards of peer review. Seven mail reviews and a written report of a
1.5-day panel review were included in the review process. Because of the
possibility of thorough discussion, the latter ensured that any issues
not addressed in writing were fully considered. The reviewers were
carefully selected to represent a variety of perspectives. Consensus was
strong, i.e., possibly unanimous. Every document indicates that the
reviewers were conscientious, thorough, and constructive. They supplied
a sound basis for NSF action. Their recommendations were heeded.

**[FINDING]{.underline}**

The integrity and efficiency of ULAFOS's merit review procedures applied
to the Unidata 2008 proposal were beyond criticism.

**A2. Implementation of NSF Review Criteria, namely intellectual merit
and broader impacts by reviewers and program officers.**

**[Mail Reviews]{.underline}**

Seven reviewers commented by mail. Five were male, two female.
Geographic distribution of reviewers was appropriate. All were at
universities and could be considered end users of Unidata products and
services, but one was at a supercomputer center and therefore could be
considered expert in the technical aspects of the proposal.

Based on reviewer feedback (6 excellent, 1 good), NSF would have been
remiss if it had not funded the proposal. The case for merit that ran
through all reviews was the absolute need for provision of earth science
(not just atmospheric science) data for both research and education. If
no Unidata system existed, it would have to be invented. Praise for
Unidata was that it was extremely forward-looking and had superbly
qualified PIs. Unidata provides far more than data. It supplies tools
for analyzing it and, perhaps most importantly, for visualizing the
enormous quantities of fundamental information now being accumulated
worldwide.

The impacts are broader than for one particular discipline; data
provided spans all the geosciences. Unidata has adopted the Earth System
Science paradigm. Its value is not confined to scientists and science
students. It is a resource for students in many general education
courses as well. Because it is accessible universally, its impact has
been especially valuable at smaller institutions and at geographically
isolated institutions.

**[Panel Review]{.underline}**

All evidence indicates the panel was exceptionally conscientious and
thorough. It spent 1.5 days meeting with Unidata personnel of all
levels. NSF Program Management participated, along with 7 external
panelists as equally divided male and female as can be achieved with an
odd number. The Panel commented that it learned a great deal more than
could possibly be provided in written reports, either in the proposal or
in the mail reviews. Hence the NSF review process filled any information
gaps that might have arisen. As the Panel itself concluded, "all the
issues raised in the course of the discussions were adequately addressed
and . . . the NSF review criteria were given due consideration."

The Panel unanimously concluded the program was excellent and deserves
to "be supported as fully as possible by NSF for the years 2003-2008."
In addition to this bottom-line conclusion, the panel offered several
findings and recommendations. The spirit in which these were presented
indicates the Panel and Unidata staff enjoyed an open exchange whose
sole purpose was to improve an already excellent program.

**[FINDING]{.underline}**

Implementation of NSF's Review Criteria for Intellectual Merit and
Broader Impacts by reviewers and program officers was exceptionally well
done.

**A3. Selection of Reviewers**

**[Mail Reviewers]{.underline}**

From the comments of the reviewers, they and their students seemed to be
users of Unidata products and services. They therefore were in a
position to provide informed judgment. One of seven reviewers was
apparently an expert in IT infrastructure. Including him was wise; since
IT is one of three pillars Unidata's proposal stands upon. Another
pillar is data presentation, especially tools that allow massive
collections of data to be presented visually in such a way that their
enormous contents can be assimilated; the third pillar is the datasets
themselves. Six reviewers, as mentioned, appeared to be users, hence
qualified to comment meaningfully. Of the six, three were with
meteorology departments, one in a department of Earth Sciences, another
in Geography, and the sixth in a School of Civil and Environmental
Science. This seems an appropriate mix.

**[FINDING]{.underline}**

The COV also notes with approval that the range of schools represented
by the reviewers spanned large to small, mainstream to isolated, the
same span Unidata intends to serve.

**[Panelists]{.underline}**

Panel chair was formerly NSF Division Director under which the Unidata
program functioned. He was qualified to note if Unidata was fulfilling
the objectives for which it was intended. The other panelists were
carefully chosen to be representative of the community in terms of
gender, size of department and institution, diverse expertise, and
geographical location.

**[FINDINGS]{.underline}**

Reviewers of the Unidata proposal were carefully and appropriately
selected. They were therefore able to provide a thoughtful evaluation of
the many aspects of products and services that Unidata proposed to
provide. They not only offered their approval of the proposal, but in
addition were able to provide useful suggestions to Unidata management.

**A4. Portfolio of awards under review: Unidata Equipment Grants
Allocation Process**

The decision to allow Unidata to allocate small equipment grants to
educational institutions that are either geographically isolated or
relatively poor in cyber-infrastructure is a wise one. It allows some of
the broader impacts NSF encourages in all its awards by partially
equalizing access to the necessities of modern research and education.
Since the amounts of the awards are small (by the standards of major
research universities but not by the universities receiving these
awards), rapid decisions are more important than prolonged analyses
aimed at precise rankings.

**[FINDING]{.underline}**

Unidata's Equipment Grant Allocation Process is appropriate. By serving
institutions outside the category of major research institutions, it
equalizes access to modern knowledge and tools that greatly enhance
those institutions' education and research.

**OTHER PROGRAMS**

The COV also reviewed the HIAPER program acquisition and the facilities
allocation process. The following observations are presented for NSF's
consideration.

**HIAPER**

The HIAPER (High-performance Instrumented Airborne Platform for
Environmental Research) represents the largest procurement to date for
UCAR and the ULAFOS organization. HIAPER is a medium sized jet aircraft
capable of carrying a sizeable scientific payload and operating in the
upper troposphere and lower stratosphere. The new platform is being
designed to serve a broad cross-section of the geosciences community. As
such the NSF has sought community input on all aspects of acquiring,
modifying and outfitting this new aircraft. An advisory panel composed
of internationally renowned atmospheric scientists, experienced in
airborne measurements, from universities, other government agencies and
NCAR was created to assure the integrity of the decision-making
processes. The advisory panel consisted of experts in many fields from
atmospheric chemistry to meteorology. In addition larger meetings (\>200
scientists) were convened to seek even broader community input on the
issues of instrumentation for the new platform. Program officers
participated in all of these community meetings and provided valuable
insight and information to the participants. The NCAR Director also
created a second Independent Review Team of other experts in both
science and project management that which reported directly to him on
the management structure and processes that NCAR was using to manage the
HIAPER Project.

**[FINDING]{.underline}**

The NSF ULAFOS management and the NCAR management are to be commended
for having sought community input from experts in both science and
management throughout the HIAPER procurement and development. By seeking
this input the NSF Program management and NCAR management have assured
that the new HIAPER platform will serve the science community well for
decades to come. The development of this facility should provide
opportunities for developing the future instrument experts needed by the
atmospheric science community. (See section C-3, item 3)

**Lower Atmospheric Observing Facilities**

**[Background]{.underline}**

LAOF provides the major research equipment for most atmospheric field
studies undertaken today. LAOF provides unique instrumentation in ground
based instruments, polarimetric radars and airborne assets. This
complement of advanced systems provides the atmospheric sciences
community its greatest advantage in observational measurements. It is
also one of the founding reasons for developing the UCAR/NCAR consortium
-- the advancement of observation equipment and platforms too expensive
to develop or support from a single university effort.

**[FACILITIES]{.underline}**

Major changes have taken place in the current facilities during the past
five years as HIAPER takes shape and the ground breaking MOA between the
NSF and the NRL provides support for the ELDORA flights.

**[Findings]{.underline}**

The COV finds the virtual radar program developed at CSU in support of
education of undergraduates in engineering and meteorology an excellent
example of progress in ULAFOS' accomplishment in educational outcomes.
Few institutions can solely support a polarimetric radar like the CHILL.
CSU has now made this system available for the education of future
engineers and scientist through the LAOF program.

The ULAFOS team is to be congratulated for these endeavors. Leadership
shown in the ULAFOS team and the equipment being developed will provide
the next generation of researchers the required tools well into the
21^st^ century. These facilities are well supported by a deployment
support group that is experienced in managing the chaotic requirements
of field programs.

This type of support for technically difficult programs with large
spatial requirements such as recent field programs as BAMEX and IHOP,
clearly demonstrate the need for the LAOF and the facilities which it
supports.

**[Recommendations]{.underline}**

The COV encourages the LAOF staff to use the CHILL radar where it can be
more cost effective in supporting missions. The CHILL and S-POL radars
are very comparable in capability. The COV urges the LAOF staff to be
more aggressive in balancing the available resources in a manner that
allocates the needed resources even if they may not be the requested
assets.

The COV notes that the T-28 is at its end-of-life, but there is still
need for a storm penetration aircraft (SPA). The COV asks the LAOF to
identify a replacement selection that will be more flexible in
supporting other missions as well as the SPA efforts. The correct
selection could relieve some of what this COV thinks will be an
over-tasking of the HIAPER asset.

**Integrity and Efficiency of the Program's Processes and Management**

The LAOF screens requirements for facilities through a review process
conducted twice yearly. Written requests are received by the NCAR and
LAOF co-chairpersons of the FAC and distributed to the committee.
Conflicts of interest are documented and members are excused from
processes where conflict of interest exists.

There are12 panel members of the OFAP with expiring terms varying from
six months to three years in groupings of two panelists. The COV has
reviewed the current committee members serving on the OFAP. It comprised
instrument designers, past and present users all knowledgeable in the
requirements and difficulty in obtaining quality field data. This panel
is most certainly capable of reviewing each request for scientific merit
and allocation of budget. Upon the completion the OFAP review is
forwarded with recommendations to the FAC for final approval.

The FAC committee, supported by co-chairs with four permanent members
and one rotating university member between CSU, SDSM&T, and University
of Wyoming reviews the OFAP findings. The FAC committee's task is to
finalize the recommendations into a final scheduling plan of support.

The panelists comprising these two committees are knowledgeable in
scientific- field operations and the associated requirements. A
numerical scoring is made by each member based on scientific merit and
the overall match of the equipment to the science goal. The co-chairs of
the FAC have resolved most conflicts of schedules before the committee
convenes through Divisional and Directorate screening. In addition,
reviews on NSF funded efforts have advantage over non-agency efforts,
but the ULAFOS takes pride in the support they provide to the nation's
research community at large.

**[Finding]{.underline}**

The COV agrees with this effort and encourages continued support to the
atmospheric community.

**[Recommendation]{.underline}**

The COV encourages the NSF to be more aggressive in government-wide
evaluation of the support facilities available for the atmospheric
sciences community and for coordinating them to improve the availability
of those assets for the atmospheric research community.

##### PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

**B.1 [NSF OUTCOME GOAL for PEOPLE]{.underline}: Developing "a diverse,
internationally competitive and globally engaged workforce of
scientists, engineers, and well-prepared citizens."**

Comments: *NSF invests in the best and brightest students, researchers
and educators to ensure a well-prepared workforce and citizenry*.\
\
To this end there are several significant accomplishments since the last
ULAFOS COV:\
\
**1**. SOARS, now in its sixth year, is designed to interest students
from traditionally underserved communities in academic degrees and
careers in atmospheric science and its related fields. It is a
year-round program that includes a ten-week paid internship each summer.
Protégés have come from over 40 participating universities and colleges.
The previous ULAFOS Panel (COV 2000) noted the exemplary contribution of
SOARS it is making toward the diversification of participation in
atmospheric science research. It pointed out that more time was required
to produce significant results. Over the past three years the effort has
begun to yield clear benefits. For example, SOARS was selected as one of
the ten institutions receiving the sixth Presidential Award for
Excellence in Science, Mathematics, and Engineering Mentoring. The award
recognizes ten institutions and ten individuals for promoting
participation among, women, minorities, and persons with disabilities in
scientific and engineering careers.

The Presidential Award goes to people and institutions who work with
students in K-12, undergraduate, or graduate-level education.

**COV Finding\
**Winning this award is testimony to the outstanding effort and
dedication that both UCAR and the NSF have shown toward reaching
underrepresented minorities. The COV hopes that this program will be
continued and extended to additional institutions to further this
important effort.

**2**.  A web site was created in summer 2001 to make available
classroom activities designed by and for teachers in collaboration with
NCAR scientists. The web site is funded by the NSF to enhance middle
school student\'s skills in science and mathematics. The project,
entitled Cycles of the Earth and Atmosphere, was created by teachers
working within Project LEARN (Laboratory Experience in Atmospheric
Research at NCAR).

The content grew out of a series of summer workshops in which 40
teachers worked with over 60 scientists from NCAR and its parent
organization, the University Corporation for Atmospheric Research
(UCAR), to create classroom-tested, hands-on activities. UCAR\'s
Cooperative Program for Operational Meteorology, Education and Training
(COMET) designed and illustrated the site. Everything on the site is
aligned with the national standards for science and math education,
which form the framework upon which most state and district standards
are being built.

**COV Finding:\
**The COV commends the NSF and UCAR for sponsoring workshops that
involve practicing teachers working together with practicing scientists
to create curriculum modules and activities for middle school students.
The middle school teachers are the experts on the standards and on the
practical aspects of what can and will work with their student\'s age
group. The scientists, working with the teachers, can help to translate
complex scientific and technical concepts into forms that will excite
young students about the Earth Sciences. These joint efforts, which
utilize the knowledge and experience of both teachers and scientists,
are the most efficient way to develop teaching tools that will help
students learn basic concepts. Web based curriculum modules are
effective ways to reach a broad populations of middle school teachers
and their students, especially in isolated situations. The web amplifies
the return on investment made in developing the teaching tools. We
applaud creating web based learning tools in order to extend the access
to additional schools and students particularly the underrepresented
population.\
\
\
**\"globally engaged workforce of scientists\"\
\
**NCAR has several scientists who serve as convening lead authors for
the Intergovernmental Panel on Climate Change (IPCC). Other NCAR
scientists are contributing to the IPCC working groups as contributing
authors.\
\
**COV Finding:\
**The IPCC is a critically important international effort to develop
consensus on the state of the knowledge of the Earth\'s Climate System.
It is extremely important that the most knowledgeable U.S. scientists,
who are recognized as leaders in their fields, participate in this
activity. It speaks well of the intellectual capital at NCAR that
several NCAR scientists participate in both leadership and contributing
author positions in this activity.  The IPCC document provides technical
information for both domestic and international environmental policy
makers and, as such, must contain the best technical information
available on the Earth\'s climate and its continual evolution.  We
commend NSF on its sponsorship of those staff engaged in this incredibly
important task.

**B.2 [NSF OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery
across the frontier of science and engineering, connected to learning,
innovation, and service to society."**

Comments: *NSF invests in ideas to provide a deep and broad fundamental
science and engineering knowledge base. It provides support for
creative, cutting edge research that yields new and important
discoveries and promotes the development of new knowledge and techniques
within and across traditional boundaries.\
\
\
***COV Finding\
\
**The NSF continues to fund research that is both scientifically
important and socially significant. Many field experiments like IHOP
contribute new understanding to weather phenomena to better predict
rainfall patterns and flash flooding.  In the climate arena work by an
NCAR scientist has provided new insight into carbon sinks, which are
vitally important to understanding climate change as trace gases
continue to accumulate in the atmosphere. Weather models developed at
NCAR have been used to provide information to incident command centers
for wildfire suppression activities in Colorado. NCAR assisted the US
Air Force in developing the limited area, mesoscale models to support
world-wide operations.

**Unidata Education Activities**

UNIDATA provides not only hardware and software to university teachers
to access near-real-time observations of the atmosphere, but also lesson
modules using such data. Universities supported by the UNIDATA program
work with local community colleges and K-12 schools to very effectively
use the "multiplier effect" in engaging the interests of students at all
levels and recruiting them into the workforce. The UNIDATA Users
Workshops outside the urban and industrial areas in the US further helps
with the introduction of the atmospheric sciences to underrepresented
groups.

Other initiatives for raising awareness on the K-12 level in the
surrounding community, for inclusion of Native American students and
tribal colleges, and for inclusion of Hispanic-serving institutions in
the lower Rio Grande region of Texas are already in the planning stages.

**COV Finding**

UNIDATA contributes significantly to the educational goals of the
National Science Foundation. ULAFOS is to be commended in its oversight
of this important program.

**B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly
accessible, state-of-the-art and shared research and education tools."**

Comments: The NCAR Facilities are used by scientists across the nation
and indeed around the world. Not only are the physical facilities in
demand, but the software facilities of climate and weather models, the
data resources and data distribution capabilities of NCAR are world
class and are widely used by the community.

**Accomplishments (additional)**

One of the expanding roles in atmospheric sciences is the understanding
of radiation processes and the effect aerosols and their associated
chemistry have on the global energy budget and public policy issues. 
Through the Ace/Asia observational study effort new understandings were
gained in how salt particles of sub-micron and super-micron sizes
influence the light extinction processes.  From this work and the
associated findings on sulfate masses the scientific community will need
to reconsider transport and radiative transfer models and the role of
sea salt in the marine-aerosol layer.

The committee acknowledges that Dr. Killeen's "Integrator Program" has
been successful in joint publications produced since his tenure began at
UCAR.

##### PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

No specific gaps noted. Suggestions for improvements have been
indicated\
at various places throughout the report.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**Participation in IGERT Competition**

NSF initiated the Integrative Graduate Education and Research
Traineeship (IGERT) program in 1997, to meet the challenges of educating
Ph. D. scientists and engineers with the multidisciplinary backgrounds
and the technical, professional, and personal skills needed to meet the
career demands of the future. The program is intended to catalyze a
cultural change in graduate education, for students, faculty, and
universities, by establishing new, innovative models for graduate
education.

Now, there are over 100 NSF sponsored IGERT programs at universities
across the country. These programs focus on problem-centered training,
delimited not by the boundaries of an academic discipline, but by the
demands of solving a problem. Combining interdisciplinary science with
practical experience, IGERT programs give their graduates the edge
needed to become leaders in their chosen fields and agents for change in
their careers. ULAFOS is the primary conduit of IGERT to the Division of
Atmospheric Sciences. This effort by ULAFOS should be continued as they
provide an opportunity for expanding the educational funding base for
ATM.

The COV believes that the IGERT program needs to be supported by the
broadest participation in the management of the program.  We did not
have any specific concerns about the current performance of ULAFOS in
the management of the activity. However, we do encourage broad
participation of the ATM Division programs to maximize interdisciplinary
input to this highly multi-disciplinary NSF program.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

NSF and UCAR manage an outstanding organization that is a leader in
atmospheric sciences and increasingly related geosciences as well.
UCAR/NCAR has earned an enviable international reputation. It is a
national treasure. One of its strengths is leadership, a focus on the
future. UCAR/NCAR therefore always evolves. The history of its evolution
has also been growth--in budgets, personnel, and fields of
investigation. Thus there exist tensions--often healthy but nevertheless
formidable\--that need to be monitored. These include:

1\. Size. Is there a maximum size of a lab that still permits effective
internal communications so that diverse activities can take advantage of
the range of skills present in the lab? Or would a few or several labs,
each concentrating on a subset of the activities, produce a better
result?

2\. Leadership vs. Service. An organization committed to meeting the
needs of university researchers must carefully weigh how it decides
which needs it fulfills. Are new facilities and services provided with
the expectation universities will need them, or are they provided
because universities demand them?

3\. Technical Capabilities. Sophisticated, complex instrumentation and
facilities are the products of talented technical people. Unless
students are trained to create them, this could become a dying art. Yet
a national center's facilities are expected to work. Therefore, the
pressure is to use only experienced, professional hard- and soft-ware
engineers. Beware. Secondly, students who never are engaged in the
development of instrumentation, its testing and calibration, tend to
accept data as accurate, whether or not it is.

4\. Mission Creep. UCAR and NCAR have such talented employees and
sophisticated infrastructure that highly competitive teams can no doubt
be put together in response to almost every new opportunity. And if
those teams succeed in the competition, it only ensures a stronger
response in the next competition. If UCAR/NCAR is to avoid
self-destructing by out-competing the universities it is their mission
to support, then they must carefully limit the span of their activities.
The COV noted that this is one area that was of concern to the Panel
reviewing the UCAR proposal in 2002. ESIG was one example that was
highlighted as being important, but UCAR and NCAR were cautioned to
focus on using the ESIG as a facilitator and integrator across the wider
academic community and not to become a major research center in its own
right.

None of these concerns were new to the ULAFOS Section at NSF. The COV is
convinced that these are topics they actively monitor and review with
UCAR management.

C.4 Please provide comments on any other issues the COV feels are
relevant.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

> The COV believes the template actually inhibits effective
> communication with the NSF management, especially for a section of the
> nature of ULAFOS. The prescribed format caused repetition and
> disconnected thoughts. A better method could have included the type of
> questions that should be covered, and allow the COV to provide an
> integrated response. The template itself had several technical flaws
> that made the final product look non-professional.

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the ULAFOS COV\]

Elbert W (Joe) Friday

Chair

[^1]: To be provided by NSF staff.

[^2]: To be provided by NSF staff.