**FY 2000 Report**

**Directorate for Geosciences (GEO)**

**Committee of Visitors for Education and Human Resource Development
(GEO-ED)**

Date of the COV: August 28-30, 2000

Program: GEO Education and Diversity programs

[Overview: 2](#overview)

[Summary Finding 2](#summary-finding)

[Process 2](#process)

[Format of the Report 3](#format-of-the-report)

[Table 1. Summary of COV findings concerning the GEO-ED Programs
4](#table-1.-summary-of-cov-findings-concerning-the-geo-ed-programs)

[Section 1 -- Individual Program Summary Findings.
5](#section-1-individual-program-summary-findings.)

[IGERT Integrative Graduate Education and Research Traineeship Program
5](#igert-integrative-graduate-education-and-research-traineeship-program)

[GK-12 7](#gk-12)

[POWRE: Professional Opportunities for Women in Research and Education
Program
9](#powre-professional-opportunities-for-women-in-research-and-education-program)

[Office of the Assistant Director Program: Global Learning and
Observations to Benefit the Environment (GLOBE).
13](#office-of-the-assistant-director-program-global-learning-and-observations-to-benefit-the-environment-globe.)

[CAREER: Faculty Early Career Development program
15](#career-faculty-early-career-development-program)

[REU: Research Experiences for Undergraduates Sites
22](#reu-research-experiences-for-undergraduates-sites)

[REU: Research Experiences for Undergraduates Supplements
26](#reu-research-experiences-for-undergraduates-supplements)

[RUI: Research in Undergraduate Institutions
28](#rui-research-in-undergraduate-institutions)

[AFGE: Awards to Facilitate Geosciences Education
32](#afge-awards-to-facilitate-geosciences-education)

[DLESE: Digital Library in Earth Science Education
36](#dlese-digital-library-in-earth-science-education)

[DIVERSITY PROGRAMS 39](#diversity-programs)

[Other (Ad-Hoc Proposals) 42](#other-ad-hoc-proposals)

[Section 2 -- COV Recommendations for AFGE Program.
45](#section-2-cov-recommendations-for-afge-program.)

[Section 3 -- COV Overview Recommendations for GEO-ED Program.
47](#section-3-cov-overview-recommendations-for-geo-ed-program.)

[Strategic Planning Recommendations
47](#strategic-planning-recommendations)

[Management and Program Review Recommendations
48](#management-and-program-review-recommendations)

#  

# Overview:

## Summary Finding

The Committee of Visitors (COV) for the Geosciences Education and Human
Resources Program (hereafter GEO-ED) met at NSF Headquarters on August
28-30, 2000. The COV finds that the GEO-ED program - while relatively
new and immature - is vibrant, healthy, and well managed. The COV
commends the program managers and particularly Dr. Mayhew, for their
achievement in creatively building the GEO-ED program within a severely
constrained resource base. The COV recommends that the Advisory
Committee for Geosciences (AC/GEO) work towards the specification and
development of a more extensive and mature education and outreach
program for GEO, building on the successes of the current program. Such
an extension should be consistent with the full recommendations of the
relevant NSF strategy documents: "NSF Geosciences Beyond 2000" and the
1997 report of the Advisory Committee for GEO "Geoscience Education: A
Recommended Strategy".

## Process

The Committee of Visitors (COV) for the Geosciences Directorate's
Education and Human Resources Program (GEO-ED) met at NSF Headquarters
on August 28-30, 2000. The review involved a detailed examination of
available jackets for each of the several individual programs making up
the overall GEO-ED effort. These include: IGERT, GK-12, POWRE, GLOBE,
CAREER, REU (Sites and Supplements), RUI, AFGE, DLESE, Diversity and
other programs, including ad-hoc proposal submissions. The committee was
asked to conduct a critical examination of process, management, and
outcomes. For the larger programs, a subset of the jackets was examined.
In these cases, the COV attempted to review a representative sampling of
funded and not-funded proposals from all three divisions in GEO. In
addition to the standard set of questions regarding the GPRA goals, the
COV was asked specifically to review and comment on the efficacy of
GEO's Awards to Facilitate Geoscience Education (AFGE) Program and on
the broad effectiveness of the overall GEO-ED program.

The charge to the committee is contained in a letter to Committee
members from Assistant Director for Geosciences Dr. Margaret Leinen
(attached).

Since these various programs are administered in differing parts of NSF
(e.g., IGERT is an agency-wide program, GLOBE is a program in the
Assistant Director's office, AFGE is a program directly administered by
GEO), the committee did not always have access to all of the relevant
jackets. However, the GEO staff had prepared a detailed dossier of
panelist and award-recipient comments that the COV found very useful in
assessing the overall program. All jackets for programs directly
administered by GEO were available for review. Other materials for the
review included a notebook of background information provided by GEO and
various helpful presentations by NSF staff, including the director of
the Division of Undergraduate Education (EHR/DUE).

## Format of the Report

Section 1 of the report summarizes the committee findings for the
various GEO-ED programs in turn. For each program, the COV has addressed
the following two issues: 1) integrity and efficiency of the program's
processes and management, and 2) outcomes and outputs of NSF's
investments (related to the GPRA requirements). Section 2 describes the
committee's findings and recommendations for the future evolution of the
AFGE program. Section 3 provides a summary of the COV's evaluation of
the overall GEO-ED program within GEO.

Table 1 summarizes the GEO-ED program funding base and the COV's primary
findings for each program element.

## Table 1. Summary of COV findings concerning the GEO-ED Programs

<table>
<colgroup>
<col style="width: 46%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 5%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 4%" />
</colgroup>
<tbody>
<tr class="odd">
<td></td>
<td><blockquote>
<p>IGERT</p>
</blockquote></td>
<td><blockquote>
<p>GK-12</p>
</blockquote></td>
<td><blockquote>
<p>POWRE</p>
</blockquote></td>
<td><blockquote>
<p>GLOBE</p>
</blockquote></td>
<td><blockquote>
<p>CAREER</p>
</blockquote></td>
<td><blockquote>
<p>REU Sites</p>
</blockquote></td>
<td><blockquote>
<p>REU Supplement</p>
</blockquote></td>
<td><blockquote>
<p>RUI</p>
</blockquote></td>
<td><blockquote>
<p>AFGE</p>
</blockquote></td>
<td><blockquote>
<p>DLESE</p>
</blockquote></td>
<td><blockquote>
<p>Diversity</p>
</blockquote></td>
<td><blockquote>
<p>Other/Ad hoc</p>
</blockquote></td>
</tr>
<tr class="even">
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td><strong>Funding for program as percentage of GEO EHR
total<sup>1</sup></strong></td>
<td>5%</td>
<td>1%</td>
<td>5%</td>
<td>3%</td>
<td>20%</td>
<td>14%</td>
<td>3%</td>
<td>14%</td>
<td>7%</td>
<td>8%</td>
<td>17%</td>
<td>3%</td>
</tr>
<tr class="even">
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td><strong>Integrity and Efficiency of Program Processes and
Management<sup>2</sup></strong></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Effectiveness of the program’s use of merit review procedures</p>
</blockquote></td>
<td>NA</td>
<td>+</td>
<td>+</td>
<td>—</td>
<td>+</td>
<td>+<sup>8</sup></td>
<td>NA</td>
<td>+</td>
<td>+<sup>11</sup></td>
<td>+</td>
<td>+</td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Use of the new NSF merit review criteria<sup>3</sup></p>
</blockquote></td>
<td>NA</td>
<td>I</td>
<td>M</td>
<td>—</td>
<td>S</td>
<td>M</td>
<td>NA</td>
<td>M</td>
<td>I</td>
<td>S</td>
<td>M</td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Reviewer selection</p>
</blockquote></td>
<td>NA</td>
<td>+</td>
<td>+</td>
<td>—</td>
<td>+<sup>6</sup></td>
<td>+</td>
<td>NA</td>
<td>+</td>
<td>+</td>
<td>+</td>
<td>+</td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Award portfolio</p>
</blockquote></td>
<td>—<sup>4</sup></td>
<td>+</td>
<td>+<sup>5</sup></td>
<td>—</td>
<td>+<sup>7</sup></td>
<td>+<sup>9</sup></td>
<td>+</td>
<td>+</td>
<td>+<sup>12</sup></td>
<td>+</td>
<td>+</td>
<td></td>
</tr>
<tr class="even">
<td><h1 id="section-1"></h1></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td><h6 id="results-outputs-and-outcomes-of-new-investments">Results:
Outputs and Outcomes of New Investments</h6></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Discoveries at and across the frontiers (GPRA Goal 1)<sup>3</sup></p>
</blockquote></td>
<td>S</td>
<td>—</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>—</td>
<td>S</td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Connections between discoveries and their use (GPRA Goal
2)<sup>3</sup></p>
</blockquote></td>
<td>—</td>
<td>—</td>
<td>M</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>M</td>
<td>M</td>
<td>—</td>
<td>S</td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Diverse, globally oriented workforce (GPRA Goal 3)<sup>3</sup></p>
</blockquote></td>
<td>—</td>
<td>—</td>
<td>S</td>
<td>—</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>M<sup>10</sup></td>
<td>S</td>
<td>—</td>
<td>S</td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Improved achievements in mathematics and science (GPRA Goal
4)<sup>3</sup></p>
</blockquote></td>
<td>—</td>
<td>—</td>
<td>M</td>
<td>S</td>
<td>S</td>
<td>—</td>
<td>S</td>
<td>M</td>
<td>S</td>
<td>—</td>
<td>S</td>
<td></td>
</tr>
</tbody>
</table>

1\. Based on summary data presented to the COV, giving approximate
average annual expenditures in each category. Note that figure given for
IGERT and GK-12 represent the internal NSF 'tax' on GEO for these
programs, not the actual expenditures on GEO programs.

2\. For categories where specific rankings were not given, + indicates a
generally positive assessment, although reference to the report text
should be made for concerns expressed; a blank entry would indicate a
generally negative assessment; NA indicates not applicable.

3\. Rankings are noted as: S=Successful; M=Minimally effective;
I=Ineffective; ---, insufficient information for the COV to evaluate.

4\. 20% of awards are cited as having some GEO involvement.

5\. POWRE funding has included 25% PIs new to NSF funding, and it is
cited by many PIs as having come at a critical, often transitional,
career stage.

6\. Some of the divisions have made great strides in ensuring
consistency and fairness in reviewing the educational aspects of CAREER
proposals. Concerns were expressed by the COV about whether the
educational aspects were receiving appropriate review.

7\. CAREER funding has a PI pool with 35% women and 9% minority. Only
15% are PIs new to NSF.

8\. The assessment of the review process was generally positive,
including the time from submission to review, but concerns were raised
about the timing of the call, the resulting timing of funding relative
to the timeline needed to get an effective program going.

9.  The award portfolio received a generally positive assessment, but
    concerns were raised about (a) continuing to ensure an appropriate
    balance between new and continuing programs, and (2) the level of
    support per student is low relative to costs and competitive
    pressures for student time.

10. There is no specific program mandate for diversity within the RUI
    program, although GEO is currently developing a Directorate-wide
    effort..

11. Documentation was often terse. It was noted that more extensive
    reviews and summary comments would provide improved feedback to PIs.

12. AFGE funding has a PI pool with 35% women and 6% minority.

# Section 1 -- Individual Program Summary Findings.

This section contains the COV's specific responses to items A and B of
its charge, namely comments on the efficiency of the grant awarding
process and on the quality of the outcomes of the program (GPRA
questions). Since several of these programs are quite new, documentation
on outcomes is not yet available. However, the COV used materials from
the dossier provided by the GEO-ED program officer to comment on
projected and interim outcomes in some cases. More general comments and
recommendations are provided in Sections 2 and 3, below.

## IGERT Integrative Graduate Education and Research Traineeship Program

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> Not applicable for this COV review -- this program is administered by
> NSF directly and not through the Geosciences Directorate. These
> jackets were not available to the COV.

2.  Effective use of new NSF Merit Review Criteria

> Not applicable -- this program is reviewed by NSF directly and not
> through the Geosciences Directorate. These jackets were not available
> to the COV.

3.  Reviewer selection

> Not applicable -- this program is reviewed by NSF directly and not
> through the Geosciences Directorate. These jackets were not available
> to the COV.

4.  Resulting portfolio of awards

> These jackets were not available to the COV. There was, however,
> information in the GEOI dossier that was reviewed by the COV. There
> are currently 75 active IGERT grants that are generally funded for a
> five year period with an average grant of \~\$500,000 annually. The
> funds are targeted for doctoral student stipends and related
> activities. The Directorate for Geosciences participates in the
> support of \~20% of the current IGERT awards. The COV did not review
> the IGERT jackets so cannot comment on the balance of projects
> relative to those submitted. There is no information on the
> participation of underrepresented groups - however, it is clear that
> many of the grants propose to target these groups (e.g. grant
> 9972810).

**1.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> **Successful:**
>
> NSF awards in this area will produce some exciting results. Grant
> 9870713, for example, will provide for the nation's first Ph.D.
> astrobiologists who will be working within an excellent
> interdisciplinary framework. Similarly, grants 9870711 and 9972817
> will generate a number of environmental engineers that will be able to
> work on "frontier" problems within urban and global scenarios.

2.  Connections between program and service to society

> Cannot be fully evaluated with the information available, though the
> COV found some very positive indications in the dossier. The enhanced
> ability for public dissemination of scientific information is
> addressed, for example, in grant 9987590 where the students will
> receive training in communication in addition to their coursework on
> human evolutionary biology.

3.  Creation of a diverse, globally-oriented workforce

> This program would appear to satisfy GPRA Goal #3. However, with no
> formal reports or summaries, there is relatively little information
> available currently on which to base this evaluation. The formal grant
> proposals suggest targeting specific populations for recruitment, but
> it is unclear as to their success rate.

4.  Improved achievements in mathematics and science

> Cannot be evaluated with the information available. Although grants
> like 9870676 clearly indicate the beginning of this process. This
> IGERT grant to Dr. Johnson will focus on applied mathematics,
> cognitive processes, and computational restraints.
>
> *Will new types of scientific databases and tools to use them be
> developed?*
>
> Yes. The bioinformatics grant (9987555) will bring together
> researchers from several fields and generate a new national genomic
> center that will focus on agriculture, biostatistics, veterinary
> medicine, and forestry.

*Will grants lead to new capabilities in education and training?*

Yes, grant 9870676, for example, considers the rapid technological
advances in the last decade and proposes to train students in computer
modeling and statistical methodology

*Will grants lead to new research on Global Change?*

Yes, there are a number of IGERT grants focusing on the environment
(9870711, 9972817, 9987590, and 9987612) that will enhance our nation's
ability to anticipate and troubleshoot potential environmental issues.

## GK-12

Three projects received funding from the GEO directorate in 1999. The
jackets were all available for review.

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> The merit review process was well documented in all jackets. The
> elapsed time to a final decision was generally 3-6 months for funded
> projects. The recommendations were very well documented. Reviewer
> questions were generally well addressed in follow-up communication
> between the program manager and the PI. All of the projects examined
> seemed to be very consistent with the goals of the GK-12 program.

2.  Effective use of new NSF Merit Review Criteria

> **Ineffective.**
>
> The reviewers did not address Criteria 1 or 2 directly. In some cases,
> Criteria 2 was addressed but, generally, reviewers only focused on the
> GK12 specific criteria.

3.  Reviewer selection

> The GK-12 proposals were reviewed by panel only. Based on the reviews
> and reviewer information, it appears that the vast majority of
> reviewers were well qualified for their assignments and came from a
> diverse group of institutions with diverse backgrounds in K-20
> education. No conflicts of interest were identified.

4.  Resulting portfolio of awards

> The quality of science and engineering is not a relevant criteria for
> these projects as they focus on teaching science through discovery.
> Awards were consistent with the objectives of the program and the
> projects proposed. No new investigators were funded although some
> relatively young investigators were. A program goal is the integration
> of education and research \-- helping researchers learn to teach and
> teachers learn to do research. All of the projects include an element
> of informal research and formal teaching experiences. All projects
> involve a higher level of risk due to their unique nature, high levels
> of collaboration among K-12, university and industry partners, and the
> large number of fellows involved.

2.  **Results: Outputs and Outcomes of NSF Investments**

> The program is too new to really assess the outcomes, however we have
> identified potential outcomes expected as a result of the projects
> funded.

1.  Discoveries at and across the frontier of science and engineering

The GK12 program will foster discoveries in our understanding of
developing collaboratives between SMET students and professionals and
K-12 teachers and students.

2.  Connections between program and service to society

> Fellowship monies will allow graduate students to continue their basic
> research while developing their teaching and communication skills. The
> fellows will be actively transferring their knowledge and research to
> the K-12 classroom through mentoring, teaching and curriculum
> development efforts. This program is a good example of how cutting
> edge science can be rapidly transferred to the general public. The new
> discoveries in how to form and nurture these partnerships can be
> easily transferred into education and policy development.

3.  Creation of a diverse, globally-oriented workforce

> The training experiences are designed to result in a more prepared
> work force capable of communicating knowledge to the public and in the
> greater understanding of science among all impacted. The GK12 projects
> funded all include an effort to serve minority and disadvantaged
> groups although this is not their primary focus. Any improvements in
> K-12 science in these environments should result in SMET careers being
> more attractive to diverse groups. Recruiting and mentoring minority
> or disadvantaged fellows provided in these programs may help them stay
> the course to meet their career goals. Having a diverse group of
> fellows may also encourage a more diverse group of students in K-12 to
> consider SMET careers.

4.  Improved achievements in mathematics and science

> The GK12 program directly addresses this need by (1) better preparing
> the next generation of SMET faculty to be educators, (2) helping K-12
> teachers better understand and teach SMET topics, and (3) making SMET
> content more engaging and relevant. Several of the projects will be
> developing educational materials that are aligned with the NSES. The
> project should result in well-trained K-12 teachers that use inquiry,
> standards-based approaches in their classrooms. The project should
> also result in advances in research on education and training through
> the assessment efforts. One of the GK12 projects is teaming with a
> local systemic initiative to improve K-12 science (DESERT Project at
> TUSD). Several of the GK12 projects will provide professional
> development to teachers.

## POWRE: Professional Opportunities for Women in Research and Education Program

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> The review process involved initial mail reviews (typically 3-5)
> followed by a panel comprised of 6-7 panelists. In some cases, mail
> reviewers were also panelists. The panel used the mail reviews to
> provide an initial ranking of the proposals. This ranking was used to
> narrow the focus of discussions on the panel, e.g., proposals
> uniformly ranked poorly were not discussed at length in the panel (for
> two of the three years reviewed) and were designated as declines.
> Through discussion, the panel revised the rankings and made
> recommendations for the fund/fund if moneys available/decline. The
> program officer in all cases awarded grants in accordance with the
> panel recommendations. Decisions were made (from submittal to award)
> in 7-9 months and documentation regarding review process was generally
> complete and clear.
>
> Mail reviews were generally focused on the intrinsic merit of the
> science proposed. Typically less than half of the mail reviewers
> addressed the POWRE criteria in their review. The panels, on the other
> hand, clearly considered both the POWRE specific priorities and
> criteria as well as scientific merit in the review and these were key
> in the final decisions. The overall review process facilitated a
> comprehensive review of all proposals in terms of scientific merit and
> POWRE specific criteria.
>
> There were significantly more proposals from EAR as compared with ATM
> and OCE and while there were more EAR awards, the success rate was
> lower than in the other two divisions in the last two years. The
> documentation did not address this issue.

2.  Effective use of new NSF Merit Review Criteria

> **Minimally Effective.**
>
> Only a handful of mail reviewers explicitly addressed Criteria 2, and
> less than half addressed the POWRE criteria which relate to Criteria
> 2. The panel explicitly used Criteria 1 and the POWRE specific
> criteria related to Criteria 2.

3.  Reviewer selection

> The number of reviews (mail) were sufficient to obtain a comprehensive
> review of the scientific merit of the proposal and provided some input
> into the POWRE and NSF Criteria 2 requirements. The panel of 6-7
> reviewers was considered adequate for a balanced review of all aspects
> of the proposals.
>
> Mail reviewers with appropriate expertise (especially regarding the
> intrinsic merit of the science) were chosen and the panel represented
> an excellent cross-section of university, government labs/centers and
> disciplines to review the broad spectrum of proposals submitted.
>
> A wide range of reviewers were involved in this program evaluation,
> primarily from universities throughout the US but also from colleges
> and government labs/agencies. Many reviewers were women. It was not
> possible to determine whether other underrepresented groups were
> involved.
>
> Documentation on the review process was complete and indicated that
> there were very few conflicts of interest but the few were
> appropriately dealt with, e.g., panelist left room during discussion
> of proposal.

4.  Resulting portfolio of awards

> There were an average of 15 awards annually for POWRE in GEO with
> total annual expenditures of approximately \$96K. The individual grant
> award size and duration were restricted by the program and awards were
> generally commensurate with these guidelines, however some awards
> exceeded the cap listed and others were for significantly smaller
> amounts. Approximately half of all awards over the past three years
> were awarded to proposals in EAR while the remaining awards were
> evenly distributed (approximately) between ATM and OCE.
>
> The grants awarded supported science and engineering research efforts
> generally characterized as excellent and very good by the reviewers.
>
> Many proposals involved utilization and exploitation of new
> technologies and methods.
>
> Approximately one quarter of the awardees were new PIs. Perhaps more
> important, this program identified and awarded grants to women at
> critical stages in their career. Many of the awardees used this grant
> to re-establish/renew/redirect research careers after
> family/administrative leaves. Others planned to use the grant to
> develop research skills and expertise, which could be then used to
> seek funding in the traditional research proposal program of NSF.
>
> The focus of many of the successful POWRE grants involved PIs learning
> new technologies and science to expand, re-establish or change their
> research program. A few funded grants were focused on developing
> educational materials and methods using new technology and research
> data. The short duration of the program and focused goals of POWRE
> limited more extensive involvement of students and curricular
> applications, although some undergraduate and graduate students were
> involved in the research projects.
>
> 100% recipients of grants were women (by program design).
> Approximately 3% were also members of other underrepresented groups.

**3.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> **Successful**
>
> The program stated objective was to increase the prominence,
> visibility and influence of women in all fields of academic science
> and engineering supported by NSF. Based on the feedback provided from
> the recipients, the program has enabled many women to establish or
> re-establish research progress and recognition in the geosciences. The
> recipients nearly unanimously believe that the grant was critical to
> the success and future of their career. There is little commentary,
> however on increases in influence/prominence of these women in the
> field or in changes in culture at the involved institutions regarding
> women.
>
> Several of the research grants have resulted in significant
> discoveries at the frontiers of science and engineering in their
> fields. For example, one investigator (NSF Grant \# 9870371), together
> with her collaborators, have developed a new biosensor based on fiber
> optic technology that can \"directly monitor microbial community
> structure and activities\... in coastal and estuarine waters and
> sediments.\" The biosensor design will allow for multiple assays in
> the future. Such a sensor should lead to new investigations and
> discoveries regarding biogeochemical processes in these environments
> including the \"relationships between cell abundance, gene expression,
> and actual microbial metabolic activities under different
> environmental conditions\".
>
> In another grant (NSF# 9806182) the PI has used a new technology which
> she and others have developed involving the synchrotron at Brookhaven
> National Lab to obtain the first in situ measurements of iron
> partitioning between differing oxidation states on minerals at micron
> resolution and has developed the \"theoretical framework\" necessary
> to better interpret the experimental results.

2.  Connections between program and service to society

> **Minimally effective.** By design the grants are limited in scope and
> duration. Subsequent connections between discoveries and service to
> society were not indicated in the few PI reports provided although
> some proposals promise such connections (e.g., NSF \# 0074635)

3.  Creation of a diverse, globally-oriented workforce

> **Successful.** By design this program serves women at critical stages
> in their career and has, based on PI testimonials, had a significant
> positive impact on their research programs. A small number of the
> women supported (3%) were also members of other underrepresented
> groups.

4.  Improved achievements in mathematics and science

> **Minimally Effective**
>
> By design the program targets the career advancement of the PIs rather
> than students or the general public. A few of the granted projects
> focused on educational development (NSF \# 9973582 and NSF \# 9973512
> & 9973243; jacket not available for this review).

*Will grants lead to new types of scientific data bases and the tools to
use them?*

One project aims to develop a geospatial data clearinghouse of marine
and coastal data and development of new research methods using GIS (NSF#
0074635).

*Will grants lead to new research on Global Change?*

Indirectly. Many grants will provide data important in accurate
understanding of our global environment.

*Will grants lead to integrative research and education opportunities at
all levels?*

A select number. For example NSF# 9973582 aims to develop student
inquiry based curricula utilizing geophysical data bases. Another PI
proposed to develop two new courses at the conclusion of the grant (NSF
#0074635). In addition, a small number of grants involved undergraduate
and graduate students in the research effort.

*Will grants lead to the participation of underrepresented groups in
integrative research & education activities?*

By definition of the program (see 3.2.3).

## Office of the Assistant Director Program: Global Learning and Observations to Benefit the Environment (GLOBE).

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

The jackets for ten 1998 GLOBE awards were not available and thus the
committee was not able to make an assessment of the merit review
procedures. The COV did not review the Inter-American Institute (IAI)
program

1.  Effectiveness of the program's use of merit review procedures

2.  Effective use of new NSF Merit Review Criteria

3.  Reviewer selection

4.  Resulting portfolio of awards

> There are ten awards jointly supported by GEO and EHR/ESIE. Each NSF
> organization contributes \$1M to the GLOBE effort. NASA, EPA, and NOAA
> are interagency partners in supporting GLOBE. NASA and EPA contribute
> \$2.5M of additional funds for the research, education, and evaluation
> components of the program. NOAA and NASA independently support the
> data archiving and visualization activities. To date, there are over
> 90 countries that have signed bilateral agreements to participate in
> the Program.
>
> The ten awards include eight grants to universities with a generally
> broad geographic distribution (including two to the University of
> Arizona) and two to NASA Centers (GSFC and Langley). Half of the
> grants are for work in the atmospheric sciences, three for
> investigations in soils and hydrology, and two for land cover and
> biology.

**4.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> **Successful.**
>
> No data are available for NSF's specific investment, however the GLOBE
> program is intended to engage students meaningfully in contributing
> data to scientific research. Students take measurements under the
> guidance of their GLOBE-trained teachers following protocols and using
> instruments specified by scientists who have a commitment to use the
> resulting data.

2.  Connections between program and service to society

> **Successful**.
>
> GLOBE has helped pioneer the use of the Web in support of K-12
> education. There is an extensive site that allows students to report
> their data to the archive and then to see visualizations of these data
> as maps and graphs which are useful in inquiry-based instruction
> world-wide.

3.  Creation of a diverse, globally-oriented workforce

> Insufficient Information to evaluate. No data are available on NSF's
> specific investment, however there are statistics on the GLOBE program
> overall suggesting that students worldwide are having positive
> experiences contributing to measurements used in earth systems
> research. To date more that 14,000 teachers from more than 9,000
> schools have been trained in GLOBE measurement protocols and use of
> GLOBE educational materials. In the United States, over one hundred
> organizations have undertaken the recruiting and training of teachers
> in GLOBE and the provision of on-going help and encouragement. To date
> over 90 countries have signed bilateral agreements to participate in
> the Program.

4.  Improved achievements in mathematics and science

> **Successful.**
>
> GLOBE addresses the standards being established for science, math,
> geography and technology education while transforming the educational
> experience for many children. Hundreds of thousands of students have
> contributed to a database of over 5 million measurements.

## CAREER: Faculty Early Career Development program

The *Faculty Early Career Development* (CAREER) Program, initiated in
FY1995, accounts for approximately 20% of GEO's expenditures for the
Education/Diversity program array in the summary figures provided to the
COV. It is the largest single program in the GEO education/diversity
portfolio. CAREER has an explicit focus on influencing the careers of
the rising generation of faculty members by encouraging integration of
research and educational efforts in their early career development.

**APPROACH TO ASSESSMENT.** Proposal jackets (\~210 total) were provided
by all GEO divisions (ATM, EAR \[with CH as a separate subset\], and
OCE) for CAREER awards and declinations. For the purposes of this
assessment, jackets were selected for review in proportion to the number
of submissions in each category (selecting \~1 of every 10) in each
division (resulting in review of 7 awards and 15 declines). All 22
jackets selected, both awards and declines, were used in answering
questions 1-3 under heading A. Summary information of CAREER awards by
principal investigator for 1998-2000, as well as the subset of award
jackets, were used in assessing question 4 under item A, and the summary
information on awards was used in conjunction with the community
responses in addressing questions 5-9 under item B.

In some cases, specific practices (e.g., about how CAREER proposals are
reviewed) are different in different divisions in GEO. The comments in
the following attempt an overall assessment for GEO, and then note any
issues particular to a given division.

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> The review mechanisms appear appropriate and rigorous for evaluating
> CAREER proposals across the divisions in GEO. This is most clear for
> the research components, as review generally follows the normal
> pathways for a research proposal, and the reviewer community and panel
> structure is experienced in this regard. All divisions solicit mail
> reviews on CAREER proposals, providing reviewers with information on
> the CAREER program with the review materials. Divisions 'reach out'
> for input from complementary NSF programs as needed on specific
> proposals.
>
> The divisions are using the experience gained from past CAREER review
> to structure and refine their review processes; the approach to
> evaluating the educational aspects has the most variety. Some
> divisions evaluate CAREER proposals only at 'regular' research panels,
> and this panel reviews both the science and the educational aspects.
> Some programs are taking a more systematic approach in recent years to
> evaluating the educational aspects. For example, OCE now evaluates the
> scientific research aspects at the 'regular' research panels (which
> often comment on the educational aspects as well), and, in addition,
> convenes a CAREER sub-panel to comment specifically on the educational
> aspects. In the most recent CAREER round, some ATM programs used two
> mail reviewers to evaluate [all]{.underline} the CAREER proposals for
> their educational content, accomplishing a similar goal as the
> sub-panel in OCE. The program review process generally balances
> evaluation of the research and educational components of the CAREER
> proposals.
>
> Decisions appear to be made efficiently and were communicated to the
> principal investigators in a timely fashion; of the jackets selected
> for review, PIs were typically notified in a 5-6 month time frame. In
> cases where final decisions came later, often because the proposal was
> 'on the bubble' for possible funding, there were interim
> communications to the PI in this regard, and these communications were
> documented in the jacket. The documentation in the jackets is
> complete, and the reasons for the final decisions are clearly defined
> by the program officers in this documentation. The decisions are
> consistent with the program's priorities and criteria. For the CAREER
> awards, both the research and the educational components had to be
> strong and balanced. The research component must be strong, and the
> educational component needs to 'go beyond' the normal expectations for
> a junior faculty member.

2.  Effective use of new NSF Merit Review Criteria

> The program is generally **successful** in using the new NSF merit
> criteria, and the combined research/educational nature of the CAREER
> proposals promoted the use of both criteria by reviewers. In many, but
> not all cases, the reviewers in all divisions explicitly commented on
> both intrinsic merit (criterion 1) and broader impact (criterion 2) in
> their review comments. Proposals that were weak on "intrinsic merit"
> (criteria 1) did not often receive a detailed evaluation of criteria
> 2, and this is viewed as appropriate. The program officers used both
> criteria in reaching decisions; in the case of the division where
> reviewers appeared least likely to divide review comments into
> subheadings for criteria 1 and 2 (OCE), the program officer's comments
> made clear and defined use of both criteria in interpreting and
> presenting the review comments.

3.  Reviewer selection

> CAREER proposals received balanced review by an appropriate number of
> reviewers. The reviewers and panel members represent an array of
> institutions, and they have strong credentials in their fields. As
> noted above, with experience, programs are refining mechanisms for
> ensuring fair and consistent review of the educational components of
> these proposals. Conflicts of interest are recognized and resolved.
>
> Reviewers are alerted to the specific nature of the CAREER program
> when receiving proposals for review. Some reviewers express dismay
> that the CAREER proposal format, with both required research and
> education components, 'shortchanges' the presentation of the science
> relative to a 'regular' NSF proposal. These potentially negative
> comments appear to be appropriately balanced by the program officers
> in using the reviews and their rankings to reach a final decision.
>
> Two competing concerns exist in regard to the educational components
> of the proposals. The first concern is the rigor with which the
> educational components are reviewed. Some programs have made great
> strides in the consistency with which the educational components are
> reviewed (this is most clear in the case of ATM and OCE). It is
> important to ensure the educational components of all GEO CAREER
> proposals receive appropriate evaluation, and this evaluation will
> need to consider the space limitations imposed by the joint nature of
> these proposals. The second, related, concern is the definition of the
> appropriate scope of the educational component, especially given the
> goal of the program in cultivating early faculty careers. The
> expectations from the review process in the jackets reviewed are that
> the work proposed must be beyond that expected in the normal course of
> faculty duties, increasingly so as the divisions, the review
> community, and the PI community gains experience with the CAREER
> program. As a community, it is in our best interest to keep the
> standard and scope of work funded in this program appropriate for
> junior faculty successful career development.

4.  Resulting portfolio of awards

> For the proposal jackets provided, the proportion of awards to total
> submissions was 28% overall. The success rate varied substantially by
> division, from a low of \~10% (EAR) to as high as \>45% (OCE). This
> may reflect the competing influences of proposal pressure, proposal
> quality, and funding availability in the divisions, as well as
> differences in how the CAREER program is viewed and promoted by the
> different divisions.
>
> The CAREER awards reviewed were all of high quality across divisions,
> and the CAREER declines reviewed were warranted, regardless of
> division. The differing success rates do not appear to reflect undue
> rejection of otherwise qualified proposals in some programs, nor do
> they reflect 'weaker' proposals being funded in some areas. The review
> processes in the different divisions appear fair and effective.
>
> The resulting portfolio of awards represents a high quality array of
> science/engineering programs, with integrated research and educational
> components as defined by the program structure itself. Awards are
> typically of longer duration than most NSF research awards, and this
> longer duration, along with flexibility and research/education
> integration, are cited as strengths by awardees. The strength of the
> longer-term support is that it builds stability for faculty career
> development, and, as a consequence, it should enable the time
> necessary for the educational innovations expected and required of
> proponents in addition to their research and normal faculty
> responsibilities. A possible drawback is that PIs do not typically
> submit new NSF proposals during their funded interval (and the
> perception is that they are discouraged from doing so in some, but not
> all programs, being considered already 'well-funded'). This leads to
> concern that they will be out of touch with the reviewer and panel
> community and out of practice in proposal submission in the strongly
> competitive environment that currently exists.
>
> Awards have gone to PIs at a variety of institutions. The funded PIs
> represent a diverse group, with women faculty constituting 35% of the
> total and minority faculty constituting 9% (1998-2000). Most
> successful CAREER PIs are already experienced NSF PIs, with 15% of
> awards to PIs new to NSF (the proportion of new PIs is increasing from
> 1998-2000).

**5.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> Successful

CAREER support to PIs leads to high quality research discoveries in all
areas supported by GEO, a clearly successful outcome of this support.
For assessing this goal, we defined 'discoveries' more broadly,
including the contributions of CAREER support in outcomes that reflect
human resource development of the PIs and of the students supported by
PIs. In the following, we comment on outcomes, as well as on specific
and general concerns.

##### General Outcome\--Better integration of research and education

> CAREER support is successful in promoting the better integration of
> research and education at the level of the individual faculty member
> supported. For example, one PI commented:
>
> "I think the largest strength of the project is the combined focus on
> a research and teaching based approach to career development. Although
> I think my proposal would have been strong enough to stand on a
> research basis only, the educational benefits of the work have already
> been excellent." \[9874751\]

##### General Outcome: PI professional development

> The CAREER program promotes the integration of research and education
> in faculty career development by awarding NSF research funds on the
> basis of a proposal, which must be strong in both aspects. Numerous
> PIs cite that academic culture as a barrier to improving education,
> i.e., the perceived or real lack of rewards:
>
> "PI would probably get more support and guidance (in terms of
> developing innovative teaching) from the department and university if
> they received encouragement from NSF." \[9507987\]
>
> "I dare say, the current structure of academia does not reward the
> enormous effort required to create strong educational programs, and
> many senior faculty give strong advice against giving time and effort
> to one's teaching career." \[9629259\]

##### Specific Outcome---Creation of new courses

> Results of CAREER awards are beginning to represent a major "agent of
> change" in curricular reform. This includes the creation of new
> courses, primarily in the spirit of inquiry- and discovery-based
> projects. Students are involved with research itself or research-like
> experiences (i.e., simulation or replication of known relations), in
> accord with major recommendations for science education reform (e.g.,
> Shaping the Future of Undergraduate SMET Education, NSF 96-139). Other
> strategies used by CAREER PIs include development of integrated
> lab/lecture courses:
>
> "The students commented that they find the lab and field experiences
> very helpful in working through the concepts they learn in the
> lecture." \[9629259\]
>
> "The \[education component\] included the development of a new
> integrated lab/lecture course\...that involves field work, in-class
> active learning exercises, and engineering case studies." \[9629259\]

##### Specific Outcome: Student research opportunities

> Virtually all of the CAREER awards for which we received PI comments
> included a component of student research opportunities, from
> undergraduate through graduate level (master's and doctoral students).
> Students are making significant contributions to overall research
> efforts, although PIs note that supervision of students does have some
> drawbacks with respect to time/resource commitments.
>
> **General Concern. Progress/accountability for educational
> components**
>
> The CAREER Programs are appropriately evaluated and funded at the
> research directorates. There is a concern that there is not sufficient
> accountability for assessing the progress of the educational
> components, both in terms of what the individual PI has accomplished,
> but, more significantly, in terms of the institutions providing their
> promised support and structure to the PI.
>
> *General Concern: Career progress of CAREER recipients*
>
> We do not have data on tenuring statistics and other monitors of
> faculty career progress for CAREER awardees, and the timeline since
> program creation is relatively short. A significant measure of the
> program's success in faculty career development will be found in such
> statistics---are CAREER PIs making professional progress at rates
> comparable to or greater than those of their peers? This is unknown.
> Given the strong competition for these awards and the rigorous
> evaluation processes, it is possible to argue that CAREER awardees
> already represent exceptionally strong junior scientists---those
> likely to succeed from the outset---so it will be important to know
> career impacts both as perceived by the individuals and by their
> institutions.
>
> General Concern. Impact of the CAREER Program in Changing the Faculty
> Reward Structure

A goal of the CAREER program is influencing the faculty reward structure
to more highly value innovation in the educational enterprise and the
integration of research and education at an early stage of faculty
careers. As noted above, we don't have the data, nor is timeline perhaps
sufficiently long to assess the impact of CAREER funding on changing the
climate of the field. Some CAREER awardees express skepticism about
how/if this change is occurring.

2.  Connections between program and service to society

#### Successful

> Value is added to the research enterprise by educational integration,
> and the CAREER program is successful in this regard. Some of the
> educational initiatives have resulted in direct and specific impacts
> in service to society:
>
> "I asked the students to take a tremendous responsibility in the final
> success of the course and design a restoration project for a local
> campus channel." \[9874751\]

3.  Creation of a diverse, globally-oriented workforce

#### Successful

> The research practices are world-class and professional, and the
> educational and research components use modern technology. The funded
> CAREER PIs are a diverse group (35% women faculty, 9% minority
> faculty), and more diverse than the typical GEO senior faculty of
> universities in the U.S. As noted above, there are not data to follow
> their career progress, nor is the impact on faculty reward structure
> known.
>
> Educational efforts include course reform and development of new
> courses at the university level, and some also involve outreach to
> underrepresented groups. These outreach efforts often have difficult
> to quantify effects, but are nevertheless valuable in providing role
> models.
>
> Although students are involved in all CAREER proposals, typically in
> the research components as well as the educational ones, recruitment
> of students especially for the research components appears to be ad
> hoc. Based on the responses received from PIs, near gender balance
> appears to have been realized. Attempts to recruit students from
> underrepresented groups have had variable success, and PIs comment on
> the reasons for this, including institutional and societal barriers.
> There does not appear to be a programmatic incentive to engage
> diversity of student participation in the research aspects, as well as
> the educational aspects, of CAREER proposals.

4.  Improved achievements in mathematics and science

#### Successful

> The educational efforts undertaken by the PIs include successful new
> courses and significant changes in how the PIs view their roles and
> responsibilities in educational efforts. However, it is hard to
> evaluate the number of students this affects. As with the career
> progress of CAREER awardees, it will take a longer time and different
> assessment methods to evaluate the influence of CAREER PIs on the goal
> of fomenting change from within geoscience departments. Educational
> efforts more closely tied to the PIs' research areas appear to be
> effective. There are some PI comments that speak to these issues:
>
> "The most significant contribution of the project was to challenge
> other atmospheric scientists to think differently about how we teach
> undergraduates meteorology. We need to offer more 'education to
> career' opportunities to these students so that they can realize that
> they can use their 'problem solving and critical thinking' skills in
> non-traditional positions that are weather-sensitive." \[9508038\]
>
> "With regard to the classroom experience, the course most directly
> connected to my research was most successful class to date."
> \[9874751\]
>
> "My course became a model around which we \[those involved in a
> division-wide re-assessment of undergraduate curriculum\] created a
> multi-disciplinary laboratory/lecture \[core curriculum, junior-year\]
> sequence." \[9629259\]

## REU: Research Experiences for Undergraduates Sites

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

```{=html}
<!-- -->
```
a)  Overall review -- The three GEO Divisions manage their REU programs
    independently. These relatively small programs are justifiably
    handled by either mail review (3 to 6 reviews) or panel review (3 to
    5 members), but not both. In EAR, in both 1998 and 1999, 5 of 11
    proposals were awarded (45%). Success rates for ATM and OCE could
    not be determined. In ATM, several program officers appear in
    jackets in the last few years. It is unclear whether proposals are
    directed to certain program officers by discipline, or if
    responsibility for the REU program is changed frequently. Lack of
    continuity may make it difficult for the community to understand the
    opportunities and process.

b)  Effectiveness of Review -- Panel reviews tend to be brief, but
    sufficient to explain the decision. Ratings were typically based on
    substantive factors, such as the mechanisms for assuring adequate
    mentoring of students and past performance of continuing programs.
    Mail reviews (ATM and OCE some years) tend to be longer, but less
    focused. While they generally offer appropriate criticisms and
    constructive suggestions, they make no explicit distinction between
    NSF's two principal criteria. No jackets for declines were available
    for ATM, so the basis for awards, beyond high ratings from mail
    reviewers, is not discernable.

c)  Efficiency -- Time to award is typically 7 -- 8 months. While this
    turnaround is reasonable, the timing of the program creates
    problems. Awards are typically made between late March and mid-May.
    REU Site programs typically begin in late May. To be effective,
    recruiting should begin no later than February 1. Clearly this
    mismatch in timing creates annual hassles for PIs. Moving the
    program deadline from Sept 15 to May 15 or earlier would alleviate
    this problem.

d)  Completeness of documentation -- Jackets are generally complete,
    though informal communications (e.g. e-mail or records of phone
    calls) may have gaps. Forms 7 for EAR are cursory but complete. They
    include a spreadsheet listing all proposals and review rankings, so
    that the status of each proposal is readily ascertained. Those in
    OCE and ATM are sufficient, but do not consistently describe the
    complete review process.

e)  Consistency with program announcement -- Reviews commonly highlight
    inconsistencies and rate proposals accordingly. One inconsistency is
    budgeting. Size of stipends and inclusion of housing and travel
    expenses are highly variable, so that requests commonly exceed the
    recommended cost per student. In part, this is because those
    recommended costs are low in the current climate of employment
    opportunities and financial need.

> **Comments**: An earlier closing date for the program would better
> match the academic year. The program should reconsider recommended per
> student costs.

1.  Effective use of new NSF Merit Review Criteria

**Minimally effective.**

Review criteria are rarely addressed explicitly. Because extraordinary
research outcomes are not expected, the focus is typically on Criterion
II.

2.  Reviewer selection

```{=html}
<!-- -->
```
a)  Number of reviewers is reasonable.

b)  Reviewers are typically respected individuals in the geoscience and
    education communities, and commonly recipients of REU or other
    education-focused awards. When panels are used, some disciplines are
    not represented. It is possible for weak science to slip by if
    logistics and pedagogy are well defined. Use of a couple of mail
    reviews by disciplinary specialists could eliminate this problem.

c)  Balance of reviewers is variable. Women are well represented, but
    underrepresented groups are rarely included, reflecting their
    scarcity in the community. Geographic balance is good for mail
    reviews, but variable for panels. The 1998 EAR panel, for example,
    consisted of three white males from the Smithsonian, University of
    Maryland, and William and Mary -- good institutional variety but
    very limited geographic distribution.

d)  Conflicts of Interest were handled appropriately.

.

1.  Resulting portfolio of awards

There are currently 36 active REU Sites: 6 in ATM, 11 in EAR, and 19 in
OCE. Quality of science is generally good, though some weaknesses may
slip by in the absence of mail reviews. The awards are typically for 3
years. Some new awards are for less time than requested to provide an
appropriate demonstration period. Some perceived excessive budgets were
trimmed. In most cases, this is justified, but the low target
per-student cost may limit recruiting in the current climate. PIs with
long-running, successful programs would appreciate longer award
durations to reduce time spent regenerating proposals.

Emerging fields are well supported. PIs applaud the program's
flexibility and openness to interdisciplinary studies. Long-running
programs may reduce opportunities for new players. These are few in
number, however, and probably appropriate. Program officers should
examine portfolios for balance of new and continuing programs and
disciplinary variety. Integration of research and education is the core
of the program, and is well supported. Recruiting of minority students
is explicitly encouraged in the program announcement, most proposals
mention it, and most reviewers applaud the efforts. Results, however,
are discouraging. Projects are commonly multidisciplinary. High risk and
innovation are relatively uncommon, and probably not a necessary
component of a successful program.

> **Additional Comments**: The size of the awards presents some
> problems. In the most recent program announcement, an increase in the
> typical stipend to "at least \$300 per week" was recommended. Even
> this may be insufficient to attract students who could earn twice that
> with other employment, and who need the income for education costs.
> The guideline of \$6000 per student total costs remained unchanged, so
> that the cost of larger stipends will have to be recouped from other
> categories.
>
> Several PIs (ATM9820590, EAR9987969, EAR9820666) noted that faculty
> spend inordinate amounts of unpaid time on logistics and secretarial
> work, yet budget limitations make it hard to squeeze in technical
> support salary. EAR9820666 also notes that institutional resources
> commonly subsidize REU Site activities and that faculty donate
> considerable time. Some reviewers protest diversion of research funds
> to support of REU student research, again putting the burden back on
> the institutions. Larger budgets would permit better support of the
> individuals who give their time to manage the programs.
>
> Several PIs note the challenge of engaging a student for a full-year
> (senior thesis?) project, given funding limits. EAR9732222 explicitly
> recommends a BREU (Beyond REU) program to allow selected students to
> continue their research. A common thread is that more well-qualified
> students could be accommodated in a larger program.

1.  **Results: Outputs and Outcomes of NSF Investments**

    1.  Discoveries at and across the frontier of science and
        engineering

> **Successful:**
>
> New discoveries are not an intended goal of the REU program. To the
> extent that students are engaged in front-line research, publishable
> outcomes are common, but truly exciting new results are relatively
> rare. Nevertheless, in several programs, students do contribute to
> acquisition of new knowledge, for example through building space
> hardware and making auroral observations.

2.  Connections between program and service to society

> **Successful**:
>
> The program engages students in the research process, thereby
> immediately incorporating results into education. To the extent that
> students take their knowledge and skills into non-academic careers
> after graduation, results are transmitted to the public and private
> sectors.

3.  Creation of a diverse, globally-oriented workforce

**Successful:**

The quality of research projects varies. In all three Divisions, several
world-class laboratories or research groups hold awards. In some cases,
however, research is rather routine, but attention to students and
pedagogical innovations are stronger. Most importantly, students with
limited research opportunities at their home institutions have the
opportunity to work with professional researchers at REU Sites.

6.2.4 Improved achievements in mathematics and science

Insufficient information to evaluate. Measures of this outcome are not
available. Testimonials, however, suggest that students engaged in
active research learn the technological and mathematical tools necessary
to do the job, probably better than in an isolated classroom setting.
Several REU Site programs invite participation by K-12 teachers, an
effective way to spread knowledge and skills to the K-12 community. This
should be encouraged.

> *Does the program effectively integrate research and education?*
>
> Integration of research and Education is a primary goal of the program
> and occurs at every site
>
> Participation of underrepresented groups in integrative research and
> education activities occurs to the extent that minorities can be
> recruited for participation. Women are well represented in most
> programs (least in ATM Sites).
>
> Professional development of teachers is achieved in the few sites that
> invite K-12 teachers as participants.

## REU: Research Experiences for Undergraduates Supplements

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> Not assessed. Awards are at the discretion of the program director.

2.  Effective use of new NSF Merit Review Criteria

Not assessed. Internal review only.

3.  Reviewer selection

> Not assessed. The supplement requests are reviewed internally.

4.  Resulting portfolio of awards

In year 2000, awards included:

> 13 ATM averaging about \$7000
>
> 31 EAR averaging about \$7300
>
> 30 OCE averaging about \$7700

These numbers are fairly typical of previous years, though perhaps a
high for OCE.

These awards constitute an integral part of some research projects. ". .
. it had been necessary to cut undergraduate support from the main
budget initially to match the level of support NSF offered."
(EAR9902898) Similarly, EAR9624501 emphasizes the value of the REU
supplements because "most of my previous projects have been cut as much
as 50%."

**7.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> **Successful:**
>
> Supplements are linked to standard research programs, many of them
> highly successful.

2.  Connections between program and service to society

**Successful:**

Training of undergraduates (and a few K-12 teachers) in current research
methods and results permits rapid transmission to society as the
participants move into careers outside of academia. One project resulted
from the PIs late realization that, using his on-going research, he
could easily create a CD-ROM of sedimentary processes for instructional
use. The short lead time for a small REU supplement allowed him to bring
students onto the project quickly to assist with assembly of the
materials.

3.  Creation of a diverse, globally-oriented workforce

**Successful:**

Participants take part in world-class research projects, typically
working as part of a team of faculty, postdocs, and graduate students.
Some projects involve international travel, exposing students to varied
cultures. The proportion of women and underrepresented minorities is
higher than the general student population. ATM9415336 states "I would
like to increase the number of these opportunities to the bright
students so that they are aware that there are other (than law and
medicine) professions worthwhile to pursue. He/She goes on to emphasize
"It does not matter to me if she decides to pursue medicine or law in
the future. At least the seed is planted in his/her mind about the
importance of climate and environment."

4.  Improved achievements in mathematics and science

**Successful:**

Students learn quantitative and technical skills in the context of the
research. "REU funds help students develop research skills early in
their careers." (EAR9900461) ". . . this salary enables students who
would normally work outside the University in the summer to do research
and interact with fellow undergraduate and graduate students."
(EAR0042377)

## RUI: Research in Undergraduate Institutions

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> EAR and ATM used the same review mechanism; no OCE proposals were
> submitted: mail review (completed by 4+ external reviewers), panel
> review, and final assessment by the Program Director. In general, the
> review process has been very fair and thorough. Proposals recommended
> for funding are systematically in the E-VG range. Successful proposals
> submitted from dominantly undergraduate institutions are uniformly
> competitive with proposals submitted from research universities as
> recognized by the quality of the science to be done. In some cases the
> panel returned more critical reviews than those received from the mail
> review. In some cases, the Program Directors reversed recommendations
> of the review panel, supporting proposals with good core ideas, but in
> need of more refinement. In general, the Program Directors have
> exercised excellent judgment in their recommendations for funding.
> They have provided clear records of their communications and
> negotiations with PIs, have worked with PIs to help address comments
> of the reviewers, particularly in the context of adjusting budgets and
> levels of effort for projects recommended for reduced funding.
> Numerous successful PIs commented on their appreciation of the level
> of communication provided by the Program Directors. Decisions to
> decline funding have been well-justified in the internal
> documentation, and in the letters of declination to the PIs. Reasons
> for declinations include limited significance of proposed work, lack
> of institutional commitment to maintain equipment, too vague research
> plans, inappropriate selection of field sites/instrumentation, lack of
> testable hypotheses, lack of knowledge of the literature, questions
> about qualifications of the researchers, and unrealistic scope of the
> project (time, effort, funding). In a few cases, proposals originally
> declined were resubmitted and received markedly higher reviews and
> recommendations for funding, indicating that the review process had
> worked in positive ways to improve the quality of the proposals. The
> time between submission and award appears to be in the range of 6-9
> months, although recommendations regarding funding generally leave GEO
> \~1 month before awards are made by DGC. The review process is in
> accord with the published guidelines, and most reviewers and PDs take
> into consideration the special conditions articulated in the
> accompanying RUI qualification letter.

2.  Effective use of new NSF Merit Review Criteria

> **Minimally effective.**
>
> The reviews have been very thorough with regard to Criterion 1, but
> there is little comment from the reviewers about Criterion 2; this
> commentary is usually limited to a note that undergraduate students
> may be involved with the research. Program Directors do comment in
> their decision justification (for awards) about student participation.
> Other aspects of Criterion 2 (e.g. diversity, impact on education
> infrastructure, outreach) are effectively ignored.

3.  Reviewer selection

> EAR and ATM have used highly credentialed scientists for both mail and
> panel reviews. These reviewers represent a variety of institutions and
> from wide geographic distribution. It was not possible to determine
> the role of underrepresented groups in the review process. Gender
> balance was not achieved, presumably reflecting the overall
> demographics of the scientific workforce. Reviewers include faculty
> from numerous Carnegie classes of academic institutions (R1,
> comprehensive), industry, state geologists, research institutes (e.g.
> marine labs), USGS, foreign scientists, and scientists from related
> disciplines (chemistry, physics, biology). Reviewers from community
> colleges were not readily evident. In general, reviews can be
> characterized as thorough, insightful, and constructively critical.
>
> One concern expressed by a successful PI concerns the perception of
> fairness in the selection of reviewers of RUI proposals:
>
> "Weaknesses include low funding levels, review of proposals by "peers"
> who may be unfamiliar with the work proposed and the difficulty of
> small liberal arts colleges or even comprehensive universities in
> obtaining funding. The last stems from the fact that most NSF proposal
> reviewers are from major research institutions and hold a bias against
> research being conducted at smaller schools".
>
> It might be good practice/policy for GEO PDs to make sure that a few
> reviews are obtained from colleagues from RUI institutions to help
> overcome this perception.

4.  Resulting portfolio of awards

> The portfolio of awards appears to track the proposal pressure of
> standard research proposals across the sub-disciplines of EAR and ATM.
> All awards are supporting the highest levels of research, as
> recognized by high evaluations by peers in the review system. The RUI
> proposals have been reviewed by the same criteria for scientific merit
> as standard research proposals, and have been recommended for funding
> that is appropriate for the scope, size, and duration of the awards.
> Most of the proposals will enhance current state of the knowledge in
> the various sub-disciplines, and a few have the potential of making
> significant new contributions. Although student participation in RUI
> research projects is viewed favorably, there does not appear to be a
> strong programmatic emphasis to engage larger educational initiatives,
> outreach programs, or recruit students from underrepresented groups

**8.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

**Successful**.

Outcomes of NSF investments include both advances in scientific
knowledge, and in the development of human resources. The program has
produced first rate science across numerous sub-disciplines in the
geosciences. These projects are commonly collaborative, typically
joining resources from many institutions (and particularly linking RUI
institutions with Research I institutions). The results have also
demonstrated applications, in some cases, to fields outside of the
geosciences (e.g., metallurgy).

> RUI projects not only produce good science, but good scientists as
> well. It has increased opportunities for well-qualified scientists who
> would not otherwise be able to competitively participate in their
> scientific fields of expertise. It also has expanded opportunities for
> students to engage research at their home institutions.

2.  Connections between program and service to society

**Minimally Effective.**

> Discoveries made through RUI funding are largely in the realm of
> "basic"science, and it is largely left to others to make applications
> to society. However, there are direct positive impacts in the human
> dimension on the development of the PIs and their students. Direct
> impacts of students who participate in the field and lab are viewed
> quite favorably by the PIs and the reviewers. Students (undergraduate
> and Master's) are typically treated as true partners in the research,
> and many have made presentations at regional and national meetings of
> professional societies, and some have co-authored refereed papers in
> peer-reviewed journals (e.g., Amer. J. Sci., Canadian Mineralogist,
> Journal of Groundwater). Many students are reported to have continued
> their careers in graduate school, as science teachers, in medical
> school, and in industry. These awards have also had a rejuvenating
> effect on the careers of the faculty PIs. One concern raised by a
> successful PI concerns the level of support of this program:
>
> "Limited funding for RUI is short sighted, results in "eating our seed
> corn". Without this support, fewer and fewer students are going to
> graduate school. "The Congress DOES seem to be worried about science
> education in America...But once the students are drawn into science,
> the support stops. It would be fair to argue that the Congress
> believes scientists should be supported at conception, but neglected
> after birth".

3.  Creation of a diverse, globally-oriented workforce

> **Minimally effective.**
>
> Diversity in participation does not appear to be a primary
> programmatic goal. Numerous PIs have reported participation by
> students from underrepresented backgrounds. It appears, overall, that
> gender balance has been achieved. However, participation by students
> of color appears to occur on an *ad hoc* basis and recruitment efforts
> have had only limited success.

4.  Improved achievements in mathematics and science

**Minimally effective.**

There is no programmatic mandate to translate research outcomes into
educational activities. A handful of students do indeed benefit from RUI
research experiences. However, results of the research are not
necessarily translated into other education or outreach activities.

## AFGE: Awards to Facilitate Geosciences Education

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> The AFGE program has relied heavily on panel reviews. This mechanism
> seems to be effective, efficient and consistent with the program
> announcement. However, the resulting documentation is very terse
> compared with the detailed mail reviews from typical NSF research
> programs. Some AFGE jackets are more extensive than the norm, and
> there is no sign that weak proposals are being funded or that the
> strongest proposals are being declined. Nevertheless, one consequence
> of the missing detail in the AFGE reviews may be that feedback to the
> proposing community is sub-optimal, e.g., PI's whose proposals have
> been declined may lack guidance as to how they can do better in the
> future.

2.  Effective use of new NSF Merit Review Criteria

> **Ineffective**.
>
> Very few of the AFGE reviewers even mention the two criteria. In
> general, the reviewers simply state the strengths and weaknesses they
> perceive in each proposal, without regard to the two criteria. The
> summaries of the panel discussions also typically do not mention the
> two criteria explicitly. The criteria may well be present in some
> tacit way in the thinking of the reviewers; if so, this is not
> apparent in the jackets. In some AFGE cases, a heavy reliance appears
> to have been placed on whether a proposal was \"catalytic\" or not,
> but this was not uniformly true.

3.  Reviewer selection

> The number of panelists has been adequate. The reviewers are certainly
> well qualified, and the panels are diverse and balanced (except in the
> K-12 area, see below). All conflicts of interest were dealt with
> appropriately during the panel discussions by having the panelist
> leave the room. There is considerable overlap between panel members
> and funded AFGE PIs, however, so that an \"AFGE community\" is clearly
> being developed. Most of the proposals are in the K-12 category in
> 1999 and 2000, and there were a large number of K-12 proposals in
> 1998. It is not clear that the panels in all those years had adequate
> K-12 expertise. Scientists and instructors from undergraduate
> universities, who appear to constitute the majority of the reviewers
> on the panels, are not generally well qualified in this respect.

4.  Resulting portfolio of awards

> As far as can be determined, the overall quality of the funded
> proposals is high. The success rate is about 25%, and the review
> process appears to be fair and objective, so it is likely that the
> awards have gone overwhelmingly to high-quality proposals. The
> following are data for each year:

**1998**

26 Awards 90 Declines Total Proposals = 116 22%

**1999**

21 Awards 47 Declines Total Proposals = 68 32%

**2000**

18 Awards 73 Declines Total Proposals = 91 20%

**For all 3 years:**

65 Awards 210 Declines Total Proposals = 275 25%

Most of the funded AFGE proposals serve the K-12 (31/65) and
undergraduate (20/65) levels. There were very few informal/public
proposals funded (6/65) and even fewer graduate and post doc level
proposals funded \-- only one in all three years of the program. Of
course, there were far fewer proposals received in these latter two
categories, especially after the first year. The breakdown of awards by
educational level is offered in the tables below. The fourth column is
the total number of proposals received for that educational level, and
the final column is the total number of proposals funded. The first
three columns report the review panel recommendations.

AFGE 1998

+--------------+--------+----------+----------+-----------+-----------+
| **Education  | **Must | **Fund   | **Do Not | **Total** | **        |
| Level**      | Fund** | if       | Fund**   |           | Proposals |
|              |        | Ava      |          | **Pr      | Funded**  |
|              |        | ilable** |          | oposals** |           |
+--------------+--------+----------+----------+-----------+-----------+
| K-12         | 9      | 0        | 33\*     | 42        | 9         |
+--------------+--------+----------+----------+-----------+-----------+
| Ugrad        | 11     | 8        | 35\*     | 54        | 11        |
+--------------+--------+----------+----------+-----------+-----------+
| Grad         | 1      | 1        | 5        | 7         | 1         |
+--------------+--------+----------+----------+-----------+-----------+
| Post-Doc     | 0      | 0        | 0        | 0         | 0         |
+--------------+--------+----------+----------+-----------+-----------+
| Multiple\*   | 0      | 0        | 0        | 0         | 0         |
+--------------+--------+----------+----------+-----------+-----------+
| Pub          | 5      | 0        | 8        | 13        | 5         |
| lic/Informal |        |          |          |           |           |
+--------------+--------+----------+----------+-----------+-----------+
| **TOTAL**    | **26** | **9**    | **81**   | **116**   | **26**    |
+--------------+--------+----------+----------+-----------+-----------+

\*These numbers are probably ± 3 due to uncertainty based on
determinations made from titles without abstracts (about half the data
were confirmed by checking jackets)

AFGE 1999

+--------------+--------+----------+----------+-----------+-----------+
| **Education  | **Must | **Fund   | **Do Not | **Total** | **        |
| Level**      | Fund** | if       | Fund**   |           | Proposals |
|              |        | Ava      |          | **Pr      | Funded**  |
|              |        | ilable** |          | oposals** |           |
+--------------+--------+----------+----------+-----------+-----------+
| K-12         | 6      | 7        | 28       | 41        | 12        |
+--------------+--------+----------+----------+-----------+-----------+
| Ugrad        | 6      | 0        | 12       | 18        | 6         |
+--------------+--------+----------+----------+-----------+-----------+
| Grad         | 0      | 0        | 3        | 3         | 0         |
+--------------+--------+----------+----------+-----------+-----------+
| Post-Doc     | 0      | 0        | 0        | 0         | 0         |
+--------------+--------+----------+----------+-----------+-----------+
| Digital      | 0      | 1        | 2        | 3         | 2         |
| Library      |        |          |          |           |           |
+--------------+--------+----------+----------+-----------+-----------+
| Pub          | 1      | 0        | 1        | 2         | 1         |
| lic/Informal |        |          |          |           |           |
+--------------+--------+----------+----------+-----------+-----------+
| **TOTAL**    | **13** | **8**    | **46**   | **67**    | **21\***  |
+--------------+--------+----------+----------+-----------+-----------+

\*This total does not count PI Prothero (U of Cal Santa Barbara:
Learning Scientific Practice in Virtual Environments) whose 1998 award
was supplemented by DUE in 1999.

AFGE 2000

+--------------+--------+----------+----------+-----------+-----------+
| **Education  | **Must | **Fund   | **Do Not | **Total** | **        |
| Level**      | Fund** | if       | Fund**   |           | Proposals |
|              |        | Ava      |          | **Pr      | Funded**  |
|              |        | ilable** |          | oposals** |           |
+--------------+--------+----------+----------+-----------+-----------+
| K-12         | 7      | 4        | 38       | 49        | 10        |
+--------------+--------+----------+----------+-----------+-----------+
| Ugrad        | 1      | 2        | 15       | 18        | 3         |
+--------------+--------+----------+----------+-----------+-----------+
| Grad         | 0      | 0        | 1        | 1         | 0         |
+--------------+--------+----------+----------+-----------+-----------+
| Post-Doc     | 0      | 0        | 0        | 0         | 0         |
+--------------+--------+----------+----------+-----------+-----------+
| Multiple     | 4      | 1        | 14       | 19        | 5\*       |
+--------------+--------+----------+----------+-----------+-----------+
| Pub          | 0      | 1        | 1        | 2         | 0         |
| lic/Informal |        |          |          |           |           |
+--------------+--------+----------+----------+-----------+-----------+
| **TOTAL**    | **12** | **8**    | **69**   | **89**    | **18**    |
+--------------+--------+----------+----------+-----------+-----------+

\*Virtually all of these proposals are related to Web-based activities
or products

> The short duration of the program to date makes it difficult to say
> whether openness to new investigators has been adequate, but there is
> no sign that the AFGE program favors any \"clique\" or \"in group.\"
> Program statistics indicate that 34% of the awards have gone to women
> and 6% to minorities. Several of the supported proposals were notably
> innovative and multidisciplinary.
>
> All in all, AFGE appears to be a valuable and healthy program, but we
> are concerned that the modest funding level (about \$1.5 million per
> year) means that many meritorious proposals go unsupported. Although
> this conclusion may seem, at first sight, to be unsupported by the
> data in the above tables, the COV believes that the final panel
> rankings were strongly influenced by the availability of funds and
> that many meritorious proposals simply fell below the funding line.
> The average award size is only about \$70K, and many awards are for
> substantially less. The small award size, the modest success rate, and
> the lack of detailed documentation of reviews are all factors that may
> contribute to discourage unsuccessful proposers from resubmitting.
> These factors may argue for an increase in the resources devoted to
> AFGE.
>
> One notable example of an AFGE award being a catalyst for further NSF
> funding is an award to M. Marlino of UCAR in 1998 entitled
> \"Multimedia Data Access System for Undergraduate Geoscience
> Education.\" This led to a proposal by a group of PIs for a Geoscience
> Education Data Library, and also to the initiation of the DLESE
> project.

**9.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> **Successful**:
>
> Based on the responses from funded PIs, the AFGE awards have succeeded
> in bridging gaps between the geoscience research and education
> communities.

2.  Connections between program and service to society

> **Minimally effective:**
>
> In many cases, the AFGE results have benefited a small community of
> users (students, educators) and have shown promise for much wider
> applicability, but insufficient time has elapsed for the products of
> the AFGE projects to have penetrated far into the potential broad user
> community. Nevertheless, there is clear evidence that many of the
> tools developed, such as Web-based educational software, clearly can
> reach exceptionally large numbers of eventual users.

3.  Creation of a diverse, globally-oriented workforce

> **Successful**:
>
> The AFGE grants have not only funded substantial numbers of women
> (34%) and significant numbers of minorities (6%) as PIs, but they have
> also in several cases targeted under-represented communities. For
> example, grant 9809837 (a partnership between the University of
> Pittsburgh and the Carnegie Museum of Natural History) contributed to
> a program including minority students. In the words of the PI, it
> reached \"a wider, younger and more diverse audience than we have in
> our university classes.\"

4.  Improved achievements in mathematics and science

> **Successful:**
>
> Although in many cases it is still too early for the projects in this
> young program to have demonstrated clear improvements in this area,
> several projects have already produced tangible results, and the
> promise of further achievements to come is obvious. For example, many
> of the AFGE projects have produced new types of scientific databases
> and tools, ranging from weather observations to revised curricula to
> educational software to research kits for students. In addition, AFGE
> scores well in having provided integrative research and training
> opportunities and providing opportunities

## DLESE: Digital Library in Earth Science Education

1.  **Integrity and Efficiency of the Program's Processes and
    Management**

    1.  Effectiveness of the program's use of merit review procedures

> There have been only two proposals that have been submitted in support
> of the implementation of DLESE \-- one in 1999 and one in 2000. In
> both cases the review mechanism was the same. These proposals were not
> taken to panel, but were subjected to extensive mail reviews (9 in
> 1999; 13 in 2000). Although overall the reviews were extremely
> positive, some specific concerns were expressed. These were
> communicated to the proponents by the Program Director, and funding
> made contingent on receipt of a substantive response to them. In the
> case of the proposal submitted in 2000 (which received 12 Excellents
> and 1 Very Good), and was in response to an RFP that specifically
> requested 1-year proposals, the Program Directors initiated adding a
> second year to the award for an amount approximately equal to the
> first year award. Their main reasons were to eliminate the need for
> the proponents to spend time and energy on submitting a second, close
> to identical, proposal the following year; to avoid another review
> process of the same proposal; and to provide the PIs with longer-term
> funding to make it easier for DLESE to attract high-quality technical
> people. However, prior to doing this, the Program Directors contacted
> the reviewers and invited comment on this action; all responded very
> favorably.
>
> This review process seems to have been very effective and included a
> large number of reviewers from academia, libraries, and software
> companies. In the case of a digital library, this is likely more
> effective than a panel review, as the breadth of expertise that can be
> tapped is greater. The process was handled in a timely manner, with
> both proposals going through this entire process and being funded in
> \~6 months. Of particular note was the willingness of the Program
> Directors to add a year of funding to a commitment that they know will
> be on-going; this is an excellent time-saving mechanism for both the
> proponents and NSF. The entire process leading up to the funding
> decision was well-documented for both proposals.
>
> There is no indication in the materials available that there have been
> any site visits. These will be an extremely important part of the
> review process as this large and complex project proceeds.

2.  Effective use of new NSF Merit Review Criteria

> **Successful:**
>
> Since these proposals grew out of a series of community-wide
> discussions (e.g. Portals to the Future, Development of a Community
> Plan) about the fundamental need for a decentralized community-based
> approach to building a digital library, many of the reviewers
> discussed both the intrinsic merit and broad impact criteria in their
> reviews.

3.  Reviewer selection

> NSF made use exclusively of mail reviews for these proposals, and
> hence requested more than typical: 19 requests for 9 responses in
> 1999, and 22 requests for 13 responses in 2000. A broad selection of
> individuals in different academic disciplines from a wide range of
> institutions, from libraries, and from software companies were asked
> to review the proposals. 40% of those requests went to women.

4.  Resulting portfolio of awards

> DLESE is in a leadership role in bringing advances in Web-based
> communications technologies to bear on building a community-based
> digital resource library. Its potential impact is huge, both in terms
> of formal and informal science education, and in outreach to the
> general public.
>
> One issue that was not addressed satisfactorily in the funding of the
> proposal is the issue of evaluation and assessment. The small
> commitments to an assessment process (\$15K in the first proposal,
> \$20K in the second proposal) were pointed out by at least one
> reviewer in the first round (1999). The \$20K in the second proposal
> is for a consultant to develop a comprehensive evaluation and
> assessment plan, and does not include any funds for implementation of
> that assessment. With a total commitment of \$5M over 5 years to this
> very complex proposal, the lack of sufficient funds allocated for
> front-end and formative assessment during this time period is a
> concern that has not been addressed by either the Program Director or
> the PIs.

2.  **Results: Outputs and Outcomes of NSF Investments**

The program is too new to really assess the outcomes. However, the COV
has reviewed the expected outcomes based on materials made available for
the review.

1.  Discoveries at and across the frontier of science and engineering

> DLESE will use a "discovery system" for navigation around its network
> that leads the user to related information. This will allow an
> individual to independently determine their own depth of learning and
> the direction that they wish their discovery experience to take.
>
> In addition, the development of DLESE is resulting in research in both
> information technology and social structures. Although building on a
> foundation of previous research, development of DLESE will result in
> advances in information technology in such areas as graphical user
> interfaces, metadata structures, and interoperability. In addition,
> DLESE will develop methods to take social commentary and translate it
> into technical reality through its governance structure within the
> DLESE Center.

2.  Connections between program and service to society

> DLESE will provide an interdisciplinary Earth Systems resource of
> high-quality materials that will allow students, educators, and the
> general public to explore the Earth, thereby enhancing not only the
> educational opportunities available to every classroom, but also the
> scientific literacy of the general public in all venues.

3.  Creation of a diverse, globally-oriented workforce

> DLESE will provide a mechanism by which the general population can be
> exposed to Earth Systems, and hence could help to entrain individuals
> into the workforce that might not have previously considered a career
> in the field. The DLESE Steering Committee has stated that a high
> priority will be to identify institutions and people with materials
> that have been effective in entraining under-represented groups, and
> will include (and highlight) those materials within their information
> system.

10.2.4 Improved achievements in mathematics and science

> DLESE will provide access to collections of high-quality, reviewed
> materials for instruction at all levels covering all components of the
> Earth system. Effectively used, these materials, together with the
> tools and interfaces that will be provided, can be expected to improve
> mathematics and science skills of all students.

## DIVERSITY PROGRAMS

A variety of programs provide funds for increasing the diversity of the
earth sciences community. Some are funded through regular GEO research
programs (SOARS and AGI Minority Scholarship Program), others through
targeted programs (CIRE), and the largest number through the REU
program. A list of funded projects that address diversity issues was
provided but it contained no information on projects that were denied
funding. Jackets included no annual or final reports to assess the
outcomes of the projects, even on projects that required annual reports.
However, the SOARS, Keck Minority REU and AGI Minority Scholarships were
renewals and the proposals contained some relevant information in the
discussion of results of prior funding.

**11.1 Integrity and Efficiency of the Program's Processes and
Management**

1.  Effectiveness of the program's use of merit review procedures

> The merit review process was well documented in all jackets. A mixture
> of review processes was used including panel and mail reviews. Time to
> decision was generally 4-6 months for funded projects. The
> recommendations were very well documented. Reviewer questions were
> generally well addressed in follow-up communication between the
> program manager and the PI. However, some projects with low or very
> low ratings were funded. All of the projects examined seemed to be
> very consistent with the specific programs funding them and the
> general goal of increasing diversity in the workforce. The ATM - CIRE
> (9872731) program is the only one that included site visits. Some
> reviewers commented that the proposal writers would benefit from some
> assistance in grant writing since they were from schools that may not
> have received a lot of grant support.

-   ATM -- CIRE 9872731: There were 56 proposals reviewed and eight
    funded. The review process consisted of 22 panelists who met and
    discussed the proposals before ranking them. The reviews focused
    primarily on the CIRE specific criteria rather than the NSF
    criteria. There was a site visit prior to agreeing to shift the
    funds from the original PI to a different PI.

-   EAR 9800619 -- mail reviews with the majority being very good or
    excellent. One 'fair' evaluation was considered by the program
    manager in the Form 7 and disregarded as a potential conflict of
    interest.

-   REU program -- These were reviewed by a panel with a primary focus
    on the REU program but those projects with a minority component were
    identified as potentially fundable by the Diversity program.

-   SOARS -- reviewed by eight reviewers with two 'fair', two 'good',
    two 'very good', and two 'excellent'. There were some valid comments
    raised by the "fair" reviewers (e.g. no documentation that a "lack
    of scholarships prevents minority students from attending graduate
    schools"). There is no tracking evident for this proposal and an
    extremely large amount of dollars invested per student compared to
    REU programs.

    1.  Effective use of new NSF Merit Review Criteria

> **Minimally effective:**
>
> Some reviewers specifically addressed NSF Criteria 1 and 2 in their
> reviews, however it was more common that reviewers indirectly
> addressed them.

2.  Reviewer selection

Each proposal had four or more mail reviews when this method was used.
Panel reviews had as many as 20 members reviewing proposals. Based on
the reviews themselves, it appears that the vast majority of reviewers
were well qualified for their assignments, came from a diverse group of
institutions with diverse backgrounds. Conflict of interest
documentation was included and conflicts were handled appropriately.
Most impressive to us was the ability of the Program Managers to
recognize and resolve reviews that were outside the consensus. In
several of the grants, the Program Manager made a concerted effort to
investigate the claims by reviewers and/or offered the PI a chance to
respond to the reviewer's comments. The "Form 7"'s were invaluable in
determining the sequence of events and the thought process involved in
the funding decision.

11.1.4 Resulting portfolio of awards

Encouragement and support for participation of underrepresented groups
is the focus of this particular group of proposals. The research to be
conducted was deemed to have high intrinsic merit by the reviewers in
all cases. However, most of the research was conducted by undergraduates
and only a small number of publications were expected. Awards were
appropriate in size and budgets were trimmed when justified. The format
of all of these programs encourages strong integration of teaching and
research and underrepresented groups were well represented.

**11.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

**Successful.**

The diversity-oriented REU experiences for undergraduates rarely result
in new and important discoveries within and across disciplinary
boundaries. However, the results, techniques used, and experiences are
very new and important discoveries for the individual participants. The
development of effective techniques for integrating education and
research in these projects, and the identification of successful
techniques or program elements that result in a more diverse work force
are very important discoveries that will move us to the frontier of
improving diversity.

2.  Connections between program and service to society

> **Successful.**
>
> Several of the research projects included environmentally-oriented
> projects or other societally-oriented investigations that allow
> students to see the connection between science and society.

3.  Creation of a diverse, globally-oriented workforce

> **Successful.**
>
> This is the main goal of the diversity programs. To a limited degree,
> these programs seem to be making progress in this area. However, the
> results of programs such as SOARS seem small compared to the
> investment.

4.  Improved achievements in mathematics and science

The research and mentoring provided in these programs is designed to
improve the level of achievement in mathematics and science. In
addition, some of the programs included a component for minority faculty
to participate in research programs, which also furthers this goal.

## Other (Ad-Hoc Proposals)

**12.1 Integrity and Efficiency of the Program's Processes and
Management**

These \'Ad Hoc\" awards are outside of formal programs although the
review process in several of them went through existing program panel
reviews. There have been a total of 24 awards over the past five years
with six declines representing an exceptional approval rating of 80%
unless there are some additional decline jackets that were not present
for the COV meeting. There were some jackets provided that were not on
the formal list of \"Ad Hoc\" awards (e.g. OCE 9814073, OCE 9907537, OCE
9616691, OCE 9841674, OCE 9909843, EAR 9614736, EAR 9727600, EAR
9840609, EAR 9809585, EAR 9900760, EHR 9820488, EAR 9901531) These 12
additional awards would increase the number of awards to 36 and the
acceptance rate to 85%. More importantly, the funding level for this
category increases from 700K to 2100K - a substantial increase!

There are six different divisions represented (EAR, DUE, ESI, OCE, HRD,
DBI) in the awards. One concern of the COV is that there is very little
to evaluate in terms of the benefits obtained for the cost. While there
are some annual and final reports in the jackets, there are no workshop
reports, videos, publications, accolades from teachers, etc. From
personal knowledge, the participants in the COV are familiar with some
of the awards and the benefits are substantial. For example, one
reviewer on OCE 9907537 voiced disapproval and concern about the typos
and errors that have been publicized on the web site. This is an
important concern that should receive priority - this is only one of the
many products that have resulted from these awards - but it is also the
only one that the COV could observe.

1.  Effectiveness of the program's use of merit review procedures

> The overall design of the process is good. The use of mail reviews and
> the panel discussions is excellent. The identification of these
> proposals as being appropriate for \"ad hoc\" funding speaks to the
> appropriateness of the review procedures. One concern that runs
> throughout the jackets is a lack of documentation of site visits -
> unless site visits were not made in which case, there is another
> possible cause for concern. While site visits would be difficult for
> the REVEL program (9814073, 9907537) during the cruises, there are
> times before and after the cruise where a NSF site visit might be very
> useful. An easy visit would be at an ASLO/AGU meeting where the
> Program Manager could visit with the minority students supported by
> 9616691, 9841674, and 9909843.
>
> All of the jackets reviewed showed that the decision time was very
> rapid and efficient. The documentation supporting the directorate's
> final decision is excellent (both for awards and declines),
> particularly where there were disparate reviews.

2.  Effective use of new NSF Merit Review Criteria

######## Minimally Effective

> There was not very good use of the new Merit Review Criteria by the
> PI\'s or the reviewers.

3.  Reviewer selection

The reviewer selection is excellent, representing a broad spectrum of
the various disciplines with an appropriate gender representation. The
expertise of the reviewers were primarily appropriate, there were
occasional reviews that came in with very high ratings without adequate
backing (e.g. EHR 9907965) although the program managers were excellent
in detecting these discrepancies and ensuring that the reviews were
weighted accordingly.

4.  Resulting portfolio of awards

> The awards are excellent and well deserving of the funds. Some of them
> represent supplements or continuations of existing projects, others
> were requesting funds for specific one-time projects (workshops,
> videos, TV programs).

**12.2 Results: Outputs and Outcomes of NSF Investments**

1.  Discoveries at and across the frontier of science and engineering

> **Successful:**
>
> This program is unusual in that the awards are for projects that
> don\'t fit into any existing programs. From reading the proposals,
> projects like EAR 0041352 (Workshop on K-12 education for
> geoscientists) and EAR 9903474 (Future research opportunities in the
> Earth Sciences: Philosophical Perspectives) clearly have a potential
> for making and disseminating discoveries at and across the frontiers.
> EAR 9809585 in particular has done an exceptional job in identifying
> both short and long term research opportunities in the earth sciences
> that have the potential to be used by NSF-EAR for strategic planning.

2.  Connections between program and service to society

> **Successful:**
>
> At least one award (EAR 0043334) is focused on the general public with
> a pilot television program.

3.  Creation of a diverse, globally-oriented workforce

> **Successful:**
>
> There are several awards such as HRD 9840962 that are targeted to
> increasing the diverse workforce. Of particular importance are the
> awards (9616691, 9841674, and 9909843) where the attendance of over
> 300 minority students at scientific meetings has resulted from the
> funds. The in-kind support (membership in ASLO) for these students is
> an excellent addition to maintaining the student\'s interest in the
> field. Two concerns addressed by reviewers should be addressed - one
> is the tracking system following these students after their experience
> and the other identifies a lack of publications from the PI regarding
> this award (10 years worth of funding thus far).

4.  Improved achievements in mathematics and science

> **Successful:**
>
> The majority of the awards address opportunities for students and
> teachers and represent educational outreach.

####### Professional development of teachers

> EAR 9727600, EAR 9840609, EAR 9911107, ESI 9911894, DUE 9953192, OCE
> 9619707, OCE 002584 all represent awards focused at teacher training
> and development. These offer exceptional opportunities for the
> teachers, particularly those that were/are able to participate in
> actual research programs. EAR 9727600 and EAR 9840609 developed and
> tested a model for the improvement of the science preparation of
> undergraduate students studying to be teachers. This is a very
> important aspect of teacher education and it is one that is almost
> completely ignored.

####### Research on Global Change

> EAR 9727631 and EAR 0041352 represent a workshop on K-12 education for
> geoscientists as part of an existing workshop at the Aspen Global
> Change Institute. Rather than focusing on research, this workshop was
> devoted to introducing the world leaders in global change research to
> education and the value of education.

# Section 2 -- COV Recommendations for AFGE Program.

The COV commends GEO for having established AFGE in direct response to
the 1997 report, *Geoscience Education: A Recommended Strategy* (NSF
97-171). It is a highly successful program that is leading to a new
community of enthusiastic geoscience education PIs and their teams. It
has generated excellent ideas, and it has already produced impressive
results. Two examples illustrate its success: (1) at the 2000 Fall AGU
Meeting, there are no less than 10 special sessions on geoscience
education; and (2) the development of the important DLESE program was
initiated, at least in part, from an AFGE project. AFGE clearly has had
a significant impact across the wide and diverse interests of geoscience
education.

The COV recommends that AFGE be continued and expanded. It plays an
important role in bridging the gap between research activities
traditionally sponsored by GEO and educational activities sponsored by
EHR. AFGE occupies a unique niche in the spectrum of educational
programs at NSF. Its proximity to the research enterprise helps to
interest researchers in contributing to educational efforts, and helps
to facilitate translation of new discoveries into classrooms and into
the public domain. One of the main strengths of AFGE resides in its
flexibility to accommodate a wide range of educational initiatives that
might otherwise "fall through the cracks" because they do not fit the
specific requirements of other focused funding programs. AFGE awards are
intended to provide "seed money" to develop prototype educational
activities, and to help participants to engage the more formal
educational programs sponsored by EHR and other funding agencies. This
will require that AFGE emphasize assessment and evaluation so that
outcomes of AFGE projects are well documented, and allow investigators
to work from a position of strength in seeking funds from new sources.

Through creative re-directing of funds, the Program Director has managed
to support high-quality proposals beyond the funds originally allocated.
As interest in becoming involved in education increases among the
scientific community, it is critical that resources grow at a rate to
permit funding of the highest quality proposals. AFGE has enjoyed a high
level of proposal pressure, which has created real challenges in
providing thorough review and feedback, and in securing sufficient funds
to award deserving proposals. The AFGE program, and the contributing
PIs, would benefit greatly through an infusion of resources to support
the review process---to ensure that the highest quality proposals
continue to be funded, and to provide formative feedback to help PIs
refine and improve their submissions. Hence, the COV recommends that
AFGE funding be approximately doubled over the next 2 years. In
addition, the potential proposal pressure and the workload on NSF staff
justify a substantial increase in the level of NSF staffing for the AFGE
program.

Just as bringing under-represented groups into the geosciences requires
bridging gaps in understanding and providing mentoring and guidance,
bringing more of the research community into the education arena
requires the same. AFGE is the component of the geoscience education
initiatives at NSF that is critical to bridging the educational-research
knowledge gap among scientists. To increase the impact of AFGE, GEO
should promote opportunities for communication among AFGE PIs and the
larger geoscience community about the essentials of effective science
education and current advances.

To further develop the AFGE program, the COV recommends that GEO
consider:

1.  supporting workshops that focus on the understanding and usage of
    fundamental concepts in effective K-12 and postsecondary education.

2.  initiating listservers and distributing AFGE progress reports to all
    AFGE PIs, and devising other community-building methods and tools to
    ensure that AFGE is more than the "sum of its parts". It is
    particularly important, in the view of the COV, that appropriate
    AFGE products receive wide distribution in the community.

3.  stressing evaluation and assessment aspects of the AFGE programs
    (with appropriate increased funding levels). Principal Investigators
    could be directed to existing resources to better utilize existing
    assessment instruments, and should also be encouraged to form
    partnerships with specialists in the fields of human learning and
    cognition.

4.  developing a GEO program designed to provide continued support for
    successfully "seeded" AFGE- programs that have demonstrable value
    for the geoscience community (this should be considered in close
    collaboration with EHR).

5.  encouraging the dissemination of instructional materials and Earth
    datasets through the DLESE collections and discovery system, and
    formation of communities of scholars through the DLESE communication
    networks

The COV was pleased to find that the GEO and EHR directorates have built
many strong collaborations, especially in the areas of undergraduate
education and instructional materials development. *These collaborations
are essential to meet the national needs in science research and science
education and should be encouraged.* The COV recommends that NSF senior
management adopt as a high priority the goal of building on these
collaborations, especially in the context of the Work Force in the
21^st^ Century initiative.

# Section 3 -- COV Overview Recommendations for GEO-ED Program.

There are many valid motivations underlying the GEO directorate\'s
program in Education and Human Resources (GEO-ED). These include
strengthening the nation\'s graduate education programs in the Earth
science as a "training ground" for future research scientists and
educators, improving K-12 and undergraduate education, and increasing
participation of historically under-represented groups in the
geosciences.

All of these objectives are vital to the well-being of the country and
indeed of the world, but perhaps the single most important reason for
mounting an excellent GEO-ED effort within GEO is to contribute toward
building science literacy in the general public. In an age when sound
science must inform wise public policy, and when individuals must deal
intelligently with a world in which the role of science is increasingly
critical, the natural window on the Earth provided by the geosciences is
key to improving widespread science literacy.

The COV has reviewed each of the individual elements comprising the
GEO-ED program within GEO. The COV finds that this program is vibrant
and healthy. The COV commends the program manager, Dr. Mayhew, and his
colleagues for their achievement in creatively building the GEO-ED
program within a severely constrained resource base.

The COV has also reviewed the relevant NSF strategy documents, including
*Geoscience Education: A Recommended Strategy* (NSF 97-171) and *NSF
Geosciences Beyond 2000* (NSF 00-27). The COV finds that the GEO-ED
program is fully consistent with these documents. However, in the
context of these strategy documents, the GEO-ED program is not yet
mature or sufficiently extensive. An implementation plan is needed to
convert the strategy into action.

The COV encourages NSF to work toward making the GEO-ED program more of
a mainstream activity within GEO and increasingly better "aligned" with
the goals and programs of the EHR directorate. The COV applauds the
diversity initiative now in the planning stage in GEO and is confident
that this initiative and the present GEO-ED program will be synergistic.

## Strategic Planning Recommendations

On the basis of its review, the COV believes that the time is ripe to
engage a planning team composed of community leaders and GEO management
staff to update and fully implement the recommendations of the report
from the Geoscience Education Working Group: "Geoscience Education: A
Recommended Strategy." Such an activity could be carried out under the
aegis of the Advisory Committee for Geosciences (AC-GEO) and could
involve extensive participation by EHR staff. The COV also believes that
GEO should be as proactive as possible in developing its posture
vis-à-vis NSF's new initiative entitled "Workforce of the 21^st^
Century" (WF). The combination of the existing successful programs
reviewed here, the new Diversity program underway within GEO, and
possible further extensions to the GEO-ED Program could provide the
underpinnings for a significant and exciting GEO contribution to the WF
program.

## Management and Program Review Recommendations

The COV task was a complex one due to the large number of programs and
jackets within the GEO-ED purview. The workload for the GEO-ED program
is extensive and the program is a multi-facetted one with significant
challenges. Based on its review, the COV makes the following specific
recommendations for future program management.

1.  GEO should consider increasing the GEO-ED administrative staff
    support to facilitate the handling of reviews and the processing of
    jackets. NSF has a stated goal of making funding decisions within 6
    months. The turn-around on GEO-ED awards is typically 7 to 9 months,
    most likely because of the very high proposal load managed by a
    single program director. Administrative funds should also be
    increased for program directors to travel to interact with the
    community, explain and promote NSF programs, and to monitor progress
    and ensure accountability for (at least) large-scale programs.

2.  GEO should carefully consider the benefits and related costs of
    using mail review -- e.g., as an augmentation for programs currently
    reviewed by panels. Scientific merit of some proposals may receive
    inadequate attention, given diverse fields and small panels.
    Inadequate feedback to unsuccessful investigators, especially in
    programs like AFGE, may result from the large number of proposals
    and panel-only review.

3.  GEO should consider ways in which the GEO-ED program can be made
    increasingly integral to the divisional and sectional activities.
    Efforts should be expended to ensure that the individual divisional
    practices with regard to agency-wide programs are discussed openly
    within GEO to provide a forum to cross-fertilize and optimize these
    practices.

4.  GEO should carefully consider the impacts on the community of
    changes in the overall portfolio of sponsored programs in the GEO-ED
    domain. Such changes can have significant impacts on proposal
    pressure levels, community expectations and, as a consequence, on
    subsequent management strategies.

5.  GEO should strive to more clearly articulate the meaning of NSF
    Merit Review Criterion 2 for education and to communicate this
    broadly to program officers, program directors, reviewers, and
    proposers.

6.  The COV recognizes a need for additional evaluation and assessment
    of GEO-ED programs, including student outcomes, program
    dissemination strategies, best practices, etc. The DLESE project,
    for example, should have a stronger evaluation component and a
    management plan that includes site visit(s). Such an effort (i.e.,
    augmented attention to evaluation and assessment) will require more
    extensive partnership with EHR/REC. The COV was pleased to hear of
    recent joint EHR/GEO-led workshops at national meetings designed to
    help the geoscience community learn more about the requirements and
    expectations of educational research programs.

7.  The GEO-ED program should continue - and augment - its efforts to
    provide detailed feedback to unsuccessful prospective PI's, so that
    "first time" proposers are not dissuaded from considering future
    participation.

8.  GEO should work to align its educational portfolio with the plans
    and strategies of the EHR directorate. In this regard, the COV
    recommends that the current, very beneficial, relationship with DUE
    be nurtured and deepened.

9.  GEO should explore new opportunities with the EHR REC and ESIE
    divisions, including the identification of programs that could
    support geoscience PI's. REC could be engaged as a partner to aid
    GEO in tracking demographics of the educational program.

10. GEO has already invested significantly in the development of DLESE,
    and this facility should be relied upon to support the broad
    portfolio of GEO educational activities. This includes dissemination
    of "best instructional practices" in all GEO disciplines and at all
    instructional levels, distribution of Earth datasets and the tools
    to use these data, and communication networks for communities of
    scholars. DLESE should also support educational activities of large
    GEO initiatives such as IRIS, UCAR, COSEE, Earthscope, and the GEO
    Diversity Program.