NSF Committee of Visitors Report

**UCAR and Lower Atmospheric Facilities Oversight Section**

**Division of Atmospheric Sciences**

**Directorate for Geosciences**

**August 1-3, 2000**

# **Response to Specific Comments in the COV Report**

Clifford A. Jacobs

Section Head

UCAR and Lower Atmospheric Facilities Section

Division of Atmospheric Sciences

October 17, 2000

We are pleased with the report of the Committee of Visitors (COV) that
met on August 1-3, 2000 to review the UCAR and Lower Atmospheric
Facilities Oversight Section (ULAFOS). There were a number of thoughtful
recommendations that will help guide the section over the next several
years.

This year the COV was asked to provide the Geosciences Directorate an
independent review of the integrity of the merit-review systems for
evaluating proposals as well as an assessment of the results of recent
ULAFOS awards. Although the COV found the generic NSF-wide charge to the
COVs not well tailored to the unique tasks of ULAFOS, there was ample
opportunity for the Committee to provide guidance and advice on a number
of topics pertinent to the Section.

With respect to the integrity of the review process and the GPRA
performance goals, I am pleased that the COV found ULAFOS was "very
successfully" meeting all the GPRA goals. The COV noted that the review
of the renewal of the NSF-UCAR cooperative agreement was conducted by
ULAFOS with integrity and thoroughness in accordance with NSF merit
review criteria and that the quality of the results funded by the
Section is excellent.

The COV report contains recommendations intended to assist ULAFOS with
future review processes, further development of programs within the
Section, and oversight responsibilities. Although these recommendations
appear throughout the report in an order consistent with the prescribed
report outline, they are summarized and grouped in a more logical way on
pages 2 and 3 of the report and will be addressed in that order.

1.  *Regarding the COV review of ULAFOS, the COV recommends tha:t*

    a.  *a tailored review process be developed for future COV review of
        ULAFOS and other sections within the Foundation where oversight
        and administration of cooperative agreements and contracts are
        major functions.*

    b.  *ATM/ULAFOS provide two types of responses in the future. First,
        a formal response to the COV report, addressed to the Advisory
        Committee for Geosciences, in which the section outlines actions
        it will take in response to the COV's findings and
        recommendations. Second, a report written for the next COV
        detailing ATM and ULAFOS actions taken and subsequent outcomes
        in response to the previous COV report.*

    c.  *the next COV pay particular attention to progress in increasing
        the participation of underrepresented groups.*

Recommendation *1.a.* is constructive and welcome. The COV process and
objectives are reviewed often with the intention of keeping the process
effective and for providing NSF with appropriate information to help it
meet its strategic and performance goals. It is recognized that for
certain Sections within NSF, such as those that provide oversight to
major facilities, the current templates and formats are not well matched
to the functions of the Sections. The GEO representatives responsible
for participation in the preparation of future COV guidelines are aware
of the difficulty the ULAFOS COV faced in addressing questions in the
report template. The ULAFOS COV experience will help in future templates
and has stressed the need for greater flexibility in reporting formats.

We agree with respect to recommendation *1.b.* The next COV will be
provided a copy of the August 2000 ULAFOS COV, this response, and a
summary of the progress made by ATM and ULAFOS towards addressing the
recommendations of the current COV.

During the review of the accomplishments of ULAFOS and ATM over the next
several years, this COV requests that the next COV pay particular
attention to progress in increasing the participation of
underrepresented groups (recommendation *1.c.*). The Directorate for
Geosciences is committed to increasing the participation of
underrepresented groups in all levels of the geoscience enterprise. It
is recognized that, although the goal is widely accepted, how to most
effectively achieve it remains under study. Nevertheless, we believe
some demonstrable progress can be shared with the next COV.

2.  *Regarding the ULAFOS review of UCAR/NCAR, the COV recommends that
    ULAFOS should (Section III-A-1:)*

    a.  *work to streamline the review process to the maximum extent
        possible, focusing narrowly on the primary goals of the review*

    b.  *carry out the management review together with the review of the
        renewal proposal. The COV recognizes that this may require
        changes to the current GEO policy with respect to the review of
        UCAR/NCAR.*

    c.  *consider alternative review and funding cycles for UCAR/NCAR to
        reduce the review burden, especially in light of UCAR/NCAR's
        40-year history of high quality work and service to the
        atmospheric science community.*

    d.  *adopt a National Academy of Sciences-style of vetting potential
        reviewers and panel members, so that potential conflicts of
        interest could be stated and documented rather than simply used
        to disqualify potential members.*

ULAFOS has worked with UCAR and NCAR management to improve the process
to be used in the next review. In addition, all that NSF, UCAR, and NCAR
learned from the last review cycle (both the positive and negative
aspects of the process) have helped refine various aspects of the
planned review process. Greatly simplified and more narrowly focused
sets of review criteria were developed as a result of interactions with
UCAR, NCAR and the UCAR Board of Trustees. The process is being
shortened and constructed to allow for the simultaneous review of
science and management aspects of the NCAR/UCAR enterprise. Both reviews
will end at approximately the same time. This approach will shorten the
period of time that the reviews will distract from the normal business
at UCAR and NCAR.

Within the Foundation, we have engaged the ATM Program Officers, the ATM
Division Director, the Assistant Director for Geosciences and the Deputy
Director and Director of NSF in a dialog on the review process. Although
a wide range of review processes were discussed, there were few that
could meet the NSF requirements for a credible, independent, and
balanced review of NCAR and the UCAR management. The Directorate for
Geosciences will, however, give this recommendation additional
consideration as we prepare for the next round and subsequent NCAR/UCAR
reviews.

Alternative review and funding cycles for NCAR and UCAR management of
NCAR have merit and are consistent with the role played by a Federally
Funded Research and Development Center (FFRDC). There are several
options that can be explored. However, options are constrained by the
Federal Acquisition Regulations (FAR), which states, \"The term of the
agreement will not exceed 5 years, but can be renewed, as a result of
periodic review, in increments not to exceed 5 years.\" For example, a
five-year Cooperative Agreement with a five-year increment after first
five years would be acceptable under FAR guidelines. This would require
a review within the first five years. This arrangement is not much
different than is planned. We will explore with NSF management and the
NSF Division of Grants and Agreements options for funding, award, and
review cycles for an FFRDC (NCAR) that are available under the FAR
regulations.

The National Academy of Sciences does not have the same review standards
and objectives as the National Science Foundation. The conflict of
interest guidelines and policies of NSF are shaped by Federal Statutes
and NSF policies and guidelines developed and refined over its 50-year
history. The integrity, thoroughness, fairness, and impartiality of the
NSF review process cannot be compromised. Potential conflicts of
interest are so important to NSF that all staff are required to
participate in an annual review and training in the conflict of interest
rules and guidelines. We do not feel that such a departure is warranted
at this time, and consider it unlikely that NSF management would approve
such a departure given the size and importance of the award to UCAR and
the importance to NSF of maintaining the integrity of the merit review
process.

3.  *Regarding a diverse workforce in science and engineering for the
    21^st^ Century, the COV recommends that ULAFOS:*

    a.  *coordinate with UCAR/NCAR, the GEO diversity effort, and
        appropriate NSF training, education and diversity activities to*

        i.  *replicate the SOARS efforts (Section III-B-7);*

        ii. *create active connections with other minority-serving
            institutions, and make better utilization of existing
            networks to increase diversity in the geosciences (Section
            III-B-7), and*

        iii. *coordinate alliances with meteorological centers and
             academic programs in developing nations (Section III-B-7).*

    b.  *collaborate with GEO, NCAR and the UCAR Consortium of
        Universities to develop an "end-to-end" strategic plan to
        recruit, employ and retain the diverse workforce of the 21^st^
        Century (Section III-B-7).*

```{=html}
<!-- -->
```
4.  *Regarding the assessment of ULAFOS-supported programs in the 21^st^
    Century workforce and the achievement in mathematics and science
    skills needed by all Americans, the COV recommends that ULAFOS work
    with UCAR/NCAR, the GEO diversity effort, and appropriate NSF
    training, education and diversity activities to:*

```{=html}
<!-- -->
```
a.  *develop long-term targets for employment of the diverse workforce
    in science and engineering, and make plans for the professional
    development and advancement of underrepresented groups in science
    and engineering (Section III-B-7);*

b.  *define metrics that quantify quality recognition of the diverse
    workforce by academia, government, business and industry (Section
    III-B-7);*

c.  *develop metrics to assess the impact of the educational programs on
    student and teacher performance, and carry out the assessment
    (Section III-B-8).*

GEO will work towards the goals articulated in recommendations 3 and 4.
We will do this in cooperation with UCAR and NCAR and the Geosciences
Diversity effort as well as the Education and Human Resources
Directorate. Since these recommendations address different aspects of
the workforce issue, the specific responses to these two recommendations
have been combined below.

The National Science Foundation Strategic Plan offers a context to
address recommendations 3 and 4. Specifically, the National Science
Board has identified five policy areas for particular attention. These
include: the Federal Investment in Science and Engineering, Educating
the National Workforce, Public Understanding and Enrichment, Science and
Engineering in a Global Context, and the assessment and improvement of
Science and Engineering Indicators. Educating the National Workforce,
Public Understanding and Enrichment, and Science and Engineering in a
Global Context are pertinent.

In addition to the NSF Strategic Plan, two activities within the NSF
purview will be helpful to ULAFOS to address the issues in
recommendations 3 and 4. NSF is charged by Congress with the
responsibility to encourage full participation of women, minorities, and
other groups currently underrepresented in scientific, engineering and
professional fields. In response to this charge NSF established the
Committee on Equal Opportunities in Science and Engineering (CEOSE).
Developing metrics to measure the progress of efforts is an important
part of the focus of this committee. The most recent report of this
committee can be found at

<http://www.nsf.gov/pubs/2000/ceose991/ceose991.html>.

The Commission on the Advancement of Women and Minorities in Science,
Engineering, and Technology Development (CAWMSET) was established by
Congress to research and recommend ways to improve the recruitment,
retention, and representation of women, underrepresented minorities
(namely, African Americans, Hispanic Americans, and American Indians),
and persons with disabilities in science, engineering, and technology
(SET) education and employment. Commission exploration of the status of
these underrepresented populations in SET has reaffirmed the Nation's
absolute economic and social imperative to ensure that all U.S. citizens
enjoy full participation at all levels of SET education and the SET
workforce. The Commission report will be an important document for
ULAFOS to use in planning a strategy for responding to this
recommendation with its partners.

*NCAR and UCAR at the Millennium: A Vision for Science, Facilities,
Service, and Leadership* articulates the major themes that dominate the
efforts of these institutions over the next several years. One of those
themes is education and training. Within this arena NCAR and UCAR have
established a record of several successful programs dedicated to
improving math and science education. These include Project LEARN
(Laboratory Experience in Atmospheric Research at NCAR), Skymath, and
the Global Change Instruction Program. Another very important activity
at UCAR is the Significant Opportunities in Atmospheric Research and
Science (SOARS), whose goal is to produce a marked increase in graduate
degrees in the atmospheric and related sciences held by underrepresented
groups. The COV recommended that this successful program be replicated.
The Geoscience Diversity initiative intends to receive proposals in the
spring of 2001. In all likelihood, these proposals will include
SOARS-like activities. In addition, other federal agencies such as DOE
have begun programs using their laboratory facilities in a manner
similar to SOARS. It is highly significant that the new Director of NCAR
intends to enhance and expand NCAR's efforts in the education and
training arena. UCAR and NCAR are keenly aware of the need to form the
right partnerships that will allow significant progress to be made in
addressing all the issues raised by the COV.

The Geoscience Diversity Program is being developed with the assistance
of Ms. Jewel Prendeville. One of the objectives of this program is to
foster connections with minority-serving institutions and make better
use of existing networks to increase diversity in the geosciences. A
diversity workshop held in August 2000, will guide the development of a
strategic plan for diversity in geosciences. One of the elements of this
plan will be the insistence of assessment metrics for the Directorate to
judge progress. These metrics will provide guidance in ULAFOS
discussions with UCAR and NCAR in assessing the impact of their
educational and outreach programs.

ULAFOS is committed to working with our internal and external partners
to address all aspects of recommendation 3 and 4. We believe the
significant amount of effort within NSF and UCAR/NCAR in addressing
these issues has helped identify the challenges, opportunities, and
strategies for making significant progress.

5)  *Regarding the optimization of NSF investment for future success,
    the COV recommends that:*

```{=html}
<!-- -->
```
a.  *ULAFOS carry out strategic planning, especially in concert with
    UCAR and NCAR (Section III-B-9);*

b.  *ULAFOS consider, in its strategic planning, strategies for
    increasing opportunities for developing a new generation of
    observation and measurement specialists (Section III-C-2);*

c.  *the GEO office work with the Biocomplexity program office to
    articulate the Geosciences themes within the broad definition of the
    Biocomplexity program (Section III-B-9);*

d.  *the ULAFOS Section Head, in cooperation with the ATM Division
    Director and the Geosciences ITR representative, work closely with
    the ITR program office to articulate the Geosciences themes in the
    ITR program (Section III-B-9).*

On October 30 and 31, 2000, NSF, UCAR, and NCAR senior staff will have a
strategic planning meeting. This meeting is the latest in a series of
strategic planning meetings dating back to 1993. NSF, UCAR, and NCAR
have found these meetings extremely productive and an excellent forum
for the exchange of ideas, strategies, and tactics for the benefit of
the entire atmospheric science community. During the upcoming meeting,
considerable discussion of education and training activities is planned.
This discussion will involve a broad range of educational and outreach
activities. The recommendations of the COV will provide additional
context for the discussions.

Facilities have traditionally occupied a significant part of the
discussion at past NSF/UCAR/NCAR retreats. The strategic planning for
facilities to serve the science needs of the community will be an
integral part of the strategy developed at this and future meetings with
UCAR/NCAR.

The next program announcement for Biocomplexity is being developed. The
Division of Atmospheric Sciences has been asked to comment on the
announcement and provide input that will help focus the program
announcement. The Division will continue to provide comment and guidance
to the Biocomplexity committee. Our focus is to provide themes within
this initiative that will allow the geoscience community to make a major
contribution.

The Information Technology Research announcement for FY 2001 has been
issued. It articulates five major scientific foci. Three are pertinent
to Geosciences, viz. Applications in Science and Engineering (AP),
Information Management (IM), and Scalable Information Infrastructure For
Pervasive Computing And Access (SI). The ULAFOS Section Head, working
with other members of the Geosciences staff, provided specific language
targeted at eliciting proposals that will synergistically improve our
knowledge and capabilities in both geoscience and information
technology.

The ITR initiative is an important component of the NSF research agenda.
It is anticipated that this program will evolve in terms of its research
focus areas and management. The ULAFOS Section Head, in cooperation with
others from GEO, will remain part of the planning and implementation
team that supports this initiative.

6)  *Regarding other areas that require attention, the COV strongly
    recommends that:*

```{=html}
<!-- -->
```
a)  *a staff person with specific expertise in Information Technology,
    and especially scientific computing and data management, be
    appointed within Geosciences (Section III-C-3),*

b)  *ULAFOS take the initiative to discuss with the responsible program
    managers the relationship of NCAR's High Altitude Observatory with
    ATM and AST/MPS, as well as with NASA's "Living with a Star"
    initiative; and that ULAFOS take a larger role in fostering HAO's
    interdisciplinary activities and its ties to the full range of Upper
    Atmosphere Section programs (Section III-C-4);*

c)  *ULAFOS take a key role in maintaining contact between UCAR/NCAR and
    the responsible NSF program managers so that all parties are fully
    informed of the details of major research and facilities initiatives
    (Section III-C-4).*

A position description has been prepared for a staff person to manage
and coordinate activities within the Directorate and across the
Foundation in the areas of information technology, scientific computing,
and data management. GEO recognizes the need for such a position and is
exploring opportunities to provide the resources necessary to satisfy
the need.

To address the concerns about the relationship of HAO, UARS, and AST
raised by the COV, ULAFOS will take advantage of the opportunity to have
key staff available during the October NSF/UCAR/NCAR planning retreat. A
meeting will be arranged among the ULAFOS Section Head, the ATM Division
Director, the Division Director of HAO, the NCAR Director, the Upper
Atmospheric Research Section (UARS), and representatives from AST/MPS.
This meeting will begin a dialog that it is hoped will lead to the
development of a mutually complementary and supportive set of activities
within NCAR, UARS, and AST. ULAFOS will continue to facilitate and
foster this dialog with a determination to improve the opportunities and
address the challenges that face the Solar-Terrestrial research agenda.

The continued communication with ATM program managers and ULAFOS staff
is essential to the success of the Section. We will continue to play a
key role in facilitating and encouraging communication between UCAR/NCAR
and NSF program managers. ULAFOS will undertake a discussion with the
Division Director and Program managers to determine areas of
communication that need improvement. In addition, ULAFOS will look for
new ways to communicate and new types of information about UCAR/NCAR
activities that the Program Managers believe would provide them with
information that will assist them in their decision making process.