**FY2003 REPORT FROM THE**

**OFFICE OF POLAR PROGRAMS**

**COMMITTEE OF VISITORS**

**Office of Polar Programs**

**National Science Foundation**

**Washington, D.C.**

**September 2003**

***[COV Report]{.underline}*** ***[annotated with OPP
responses]{.underline}*** ***[following each
recommendation.]{.underline}***

**FY 2003 REPORT FROM THE OFFICE OF POLAR PROGRAMS**

**COMMITTEE OF VISITORS (COV)**

**17-19 SEPTEMBER 2003**

Table of Contents

[NSF Committee of Visitors (COV) Reviews
3](#nsf-committee-of-visitors-cov-reviews)

[NSF Office of Polar Programs FY 2003 COV Review of the
3](#nsf-office-of-polar-programs-fy-2003-cov-review-of-the)

[Arctic Science Section and Antarctic Science Section
3](#arctic-science-section-and-antarctic-science-section)

[The FY 2003 OPP Committee of Visitors (COV)
3](#the-fy-2003-opp-committee-of-visitors-cov)

[The Agenda and Work plan of the FY 2003 Committee of Visitors
4](#the-agenda-and-work-plan-of-the-fy-2003-committee-of-visitors)

[Sources of information and data for the period FY 2000-2002
4](#sources-of-information-and-data-for-the-period-fy-2000-2002)

[Review of proposal jackets 5](#review-of-proposal-jackets)

[Responses to NSF Committee of Visitors Core Questions
5](#responses-to-nsf-committee-of-visitors-core-questions)

[A.1 Quality and Effectiveness of Merit Review Procedures
6](#a.1-quality-and-effectiveness-of-merit-review-procedures)

[A.2. Implementation of NSF Merit Review Criteria
8](#a.2.-implementation-of-nsf-merit-review-criteria)

[A.3. Selection of Reviewers 9](#a.3.-selection-of-reviewers)

[A.4. Resulting Portfolio of Awards
9](#a.4.-resulting-portfolio-of-awards)

[A.5. Management of the Program Under Review
13](#a.5.-management-of-the-program-under-review)

[B.1 NSF Outcome Goal for People: Developing "a diverse internationally
competitive and globally engaged workforce of scientists, engineers, and
well-prepared citizens."
15](#b.1-nsf-outcome-goal-for-people-developing-a-diverse-internationally-competitive-and-globally-engaged-workforce-of-scientists-engineers-and-well-prepared-citizens.)

[B.2 NSF Outcome Goal for Ideas: Enabling "discovery across the frontier
of science and engineering, connected to learning, innovation, and
service to society."
15](#b.2-nsf-outcome-goal-for-ideas-enabling-discovery-across-the-frontier-of-science-and-engineering-connected-to-learning-innovation-and-service-to-society.)

[B.3 NSF Outcome Goal for Tools: Providing "broadly accessible,
state-of-the-art and shared research and education tools."
16](#b.3-nsf-outcome-goal-for-tools-providing-broadly-accessible-state-of-the-art-and-shared-research-and-education-tools.)

[C.1 Program areas in need of improvements or gaps within program areas
16](#c.1-program-areas-in-need-of-improvements-or-gaps-within-program-areas)

[C.2 Program performance in meeting program-specific goals and
objectives not covered by Core Questions A1-29 and B1-3
17](#c.2-program-performance-in-meeting-program-specific-goals-and-objectives-not-covered-by-core-questions-a1-29-and-b1-3)

[C.3 Agency wide issues that should be addressed by NSF to improve the
program's performance
17](#c.3-agency-wide-issues-that-should-be-addressed-by-nsf-to-improve-the-programs-performance)

[C.4 Other relevant issues 18](#c.4-other-relevant-issues)

[C.5 Improving the COV review process, format and report template
20](#c.5-improving-the-cov-review-process-format-and-report-template)

**FY 2003 REPORT FROM THE OFFICE OF POLAR PROGRAMS**

**COMMITTEE OF VISITORS, 17-19 SEPTEMBER 2003**

# NSF Committee of Visitors (COV) Reviews

An NSF Committee of Visitors (COV) is asked to provide "a balanced
assessment of ..... performance in two primary areas, the integrity and
efficiency of the *processes* related to proposal review, and the
quality of the *results* of investments in the form of outputs that
appear over time. The COV also explores the relationships between award
decisions and program/NSF-wide goals in order to determine the
likelihood that the portfolio will lead to the desired results in the
future ....... It is important to recognize that reports generated by
COV's are used in assessing agency progress in order to meet
government-wide performance reporting requirements, and are made
available to the public...." (*Committee of Visitors Reviews, NSF Manual
1, Section VII*).

COV's tasked with reviews of NSF Divisions, Directorates and Offices in
2003 are asked to respond to a set of Core Questions organized within
the following major categories:

A.1. Quality and Effectiveness of Merit Review Procedures

A.2. Implementation of NSF Merit Review Criteria

A.3. Selection of Reviewers

A.4. Resulting Portfolio of Awards

A.5. Management of Program Under Review

B.1 NSF Outcome Goal for People

B.2 NSF Outcome Goal for Ideas

B.3 NSF Outcome Goal for Tools

C.1 -- C.5 Other issues that the COV feels are relevant to the review

# 

# NSF Office of Polar Programs FY 2003 COV Review of the

# Arctic Science Section and Antarctic Science Section

This report presents the results of the FY 2003 COV review of the
Antarctic Science and Arctic Science (including logistics) sections of
the Office of Polar Programs for the period FY 2000-2002, and it follows
the template of major topic areas and Core Questions set forth in the
NSF COV directive cited above. It is noted that these sections were last
reviewed in July of 2000 (spanning the period FY 1997-1999).

**In summary, the COV finds OPP to be effectively managed, with proposal
solicitation and review increasingly addressing both major review
criteria by the end of the period reviewed. Importantly, the results of
OPP's investments are exciting, worthwhile and of high quality, and
OPP's administrative and management processes are thorough and sound,
with high integrity.**

# The FY 2003 OPP Committee of Visitors (COV)

Dr. Karl Erb, Director of the Office of Polar Programs, appointed a
Committee of Visitors (COV) comprising: Raymond Bradley (University of
Massachusetts), Howard Epstein (University of Virginia), Sven Haakanson
(Alutiiq Museum, Kodiak), Beverly Hartline (Argonne National
Laboratory), Gonzalo Hernandez (University of Washington), Martin
Jeffries (University of Alaska, Fairbanks), Molly Miller (Vanderbilt
University), Marilyn Raphael (University of California, Los Angeles),
James Swift (Scripps Institution of Oceanography, La Jolla), James
Taranik (University of Nevada, Reno), Peter Webb (Ohio State
University), and Karen Wishner (University of Rhode Island). The
committee was chaired by Peter Webb, with Beverly Hartline and Martin
Jeffries representing the Office of Polar Programs Office Advisory
Committee (OAC). The Committee of Visitors (COV) is an *ad hoc*
subcommittee of the OAC. The expertise among the FY 2003 COV group
spanned most specialty areas in the OPP's science programs.

# 

# The Agenda and Work plan of the FY 2003 Committee of Visitors 

The FY 2003 Committee of Visitors met at the National Science Foundation
over three days from 17-19 September 2003, during Hurricane Isabel.

Dr. Erb presented the charge to the FY 2003 Committee of Visitors and
clarified its duties within the framework of NSF's Core Question
template. To launch the task and assist the committee, the meeting
commenced with overview presentations by senior Office of Polar Program
administrative staff. Dr. Karl Erb provided a comprehensive overview of
the OPP mission, this including information on the current Office
administrative structure and personnel, major areas of science
administered by OPP in both the Arctic and Antarctic regions, the role
of OPP in promoting NSF agency-wide priorities in research, education
and technology, the representation of polar science to the public and
society at large, proposal, budget and other information. Dr. Robert
Wharton briefed the committee on Conflict of Interest issues. Then
followed illustrated presentations of past, planned and proposed OPP
program activities within the Arctic Science Section (Dr. Thomas Pyle,
Section Head) and Antarctic Science Section (Dr. Scott Borg, Section
Head).

During the following two and a half days the COV considered data from
these presentations along with a range of other of documentation as it
addressed NSF Core Questions and developed summary comments and
recommendations. The onset of Hurricane Isabel and the consequent
closing of government offices prevented the COV from discussing its
conclusions with OPP program officers and managers at the end of its
deliberations. However, the committee chair and one other COV member met
with the OPP staff the following week for this purpose. The work of the
committee was rendered more difficult by the government closing but
arrangements were made for it to complete much of its activity in a
local hotel. Final report editing was conducted via e-mail and phone
discussions.

# 

# Sources of information and data for the period FY 2000-2002

Responses to Core Questions together with summary comments and
recommendations provided below are based on the following sources of
information.

1\. Program officer briefings and questioning.

2\. Proposal jackets (proposal, mail reviews, panel reviews, program
manager statements, correspondence, award letters, annual reports, etc).

3\. Office of Polar Programs and NSF Electronic Information System (EIS)
spread sheet data.

a\. Dwell times for awarded and declined proposals

b\. Award dollar amounts

c\. Award duration

d\. Numbers of new principal investigators

e\. Funding (award) rates for underrepresented groups (minorities and
women)

f\. Funding (award) rates for principal investigators by program
specialties

g\. Funding (award) rates by geographic region

h\. Funding (award) levels by Carnegie institutional category

i\. Types of proposal review (mail and/or panel)

NSF FY 2003-2008 GPRA Strategic Plan (Draft 3.1, NSB-03-70 (June 5^th^
2003).

NSF Office of Polar Programs Advisory Committee (OAC) Report on GPRA

(November 2000).

NSF Office of Polar Programs Advisory Committee (OAC) Report on GPRA

(November, 2001).

NSF Office of Polar Programs, Government Performance and Results Act
(GPRA)

FY 2002 Performance Report (2002).

NSF OPP Advisory Committee, Working Group on Implementation of Review

Criterion #2 "Broader Impacts of the Proposed Study," Merit Review

Criteria (February 2001)

FY 2000 Report from the Office of Polar Programs Committee of Visitors

(25-27 July 2000); and OPP Response to Recommendations of the FY 2000
OPP COV (July 25-27, 2000).

The United States in Antarctica, Report of the U.S. Antarctic Program
External Panel, U.S. National Science Foundation, Washington, D.C.
(April, 1997).

Polar Press Clips 2003, U.S. National Science Foundation, Washington,
D.C.

List of review criteria for the solicitations and program announcements
issued during the period under review (2000-2002).

# 

# Review of proposal jackets

Proposal jackets provided a major source of information used by the
Committee of Visitors in addressing the NSF Core Questions. The
committee examined a total of 176 proposal jackets from the period
2000-2002 during its survey.

Seventy-four (74) and one hundred and two (102) jackets fell within the
"awarded" and "declined" categories respectively. Proposal jackets
surveyed were randomly selected under Dr. Erb's direction using a random
number generator to select 10% of the proposal actions (awards and
declines) from each program under review. In the "awarded" category
there was the following distribution: Arctic Science Section (36),
Antarctic Science Section (36), and General instrumentation (2). In the
"declined" category there was the following distribution: Arctic Science
Section (60), Antarctic (38), and General Instrumentation (4). Ten SGER
proposals were included in the total sample. Every jacket in the sample
was reviewed by several COV members.

In its review of jackets and other material, the COV addressed the
nearly 40 Core Questions provided in NSF's standard guidance to COV's.
Given the consistency of the material on which we based our conclusions,
we believe they are unlikely to be affected by either a more exhaustive
examination of the available proposal jacket sample or by consideration
of a larger jacket sample.

# 

# Responses to NSF Committee of Visitors Core Questions

The following sections present committee responses to specific Core
Questions.

# A.1 Quality and Effectiveness of Merit Review Procedures

***1. Is the review mechanism appropriate? (panels, ad hoc reviews, site
visits)***

Yes. OPP uses both mail reviews and panel reviews to evaluate proposals,
and obtains a minimum of three reviews for each proposal. We found no
instances of site visits being used. Absence of site visits is
appropriate, since the proposals examined by the COV were not for
centers or institutes.

Using their particular expertise and experience, mail reviewers provide
detailed evaluations of individual proposals. Panels provide collective
evaluations and comparisons among several proposals, synthesizing
assessments using the collective experience and expertise of the panel
members. Panel review reports appear not to contain as much detail as
three or more mail reviews do together.

In both the Arctic and Antarctic sections mail reviews and panel reviews
are used appropriately, in ways that reflect the nature and scope of
individual programs. For example, the Arctic Natural Sciences Program,
which receives more proposals than any other OPP program, uses primarily
mail reviews. This is not surprising, as it would be impractical for
this multidisciplinary program to assemble a panel with the necessary
disciplinary depth and breadth to provide an effective review of each
proposal. On the other hand, it is appropriate for the Arctic System
Science program to rely exclusively on panel reviews to assess proposals
submitted in response to special announcements and requests for
proposals such as the "Freshwater Initiative." Given the special
strengths of mail and panel reviews, we believe that OPP program
managers should continue to be flexible and use their discretion to
employ the most appropriate review mechanism, and consider using both
types of review together, whenever doing so would be valuable.

***2. Is the review process efficient and effective?***

Yes. The review process and subsequent communication of decisions to
principal investigators was found to be generally good. Reviews of
proposals cannot be returned to PIs until a final decision is made by
the Section Head. The COV stresses the importance of returning reviews
as quickly as possible in cases where a proposal is declined so that a
PI might re-write a proposal and re-submit in time for a subsequent
submission deadline. We recognize that award notifications may be
delayed by logistic and budget-related deliberations.

[Recommendation: Declination letters, including access to the reviews,
should proceed on as fast a track as possible, in order to
allow]{.underline} [timely submission of revised proposals.]{.underline}

**OPP Response**: OPP agrees that declination decisions should be
relayed to proposers as quickly as possible, and will take steps to
expedite such communications. Apart from those few instances where
decisions must await final information about program budgets or
logistics limitations, declinations will be processed on a separate fast
track. Ways to achieve this goal will be a topic of discussion at the
retreat mentioned above.

***3.** **Are reviews consistent with priorities and criteria stated in
the program solicitations, announcements, and guidelines***

Mostly. The reviews in nearly all cases seem to be consistent with the
broad nature of the solicitations. With respect to the two major
guidelines for NSF proposal review, Intellectual Merit and Broader
Impacts, the Intellectual Merit of the proposal was in all cases
addressed consistent with the priorities and criteria stated in the
solicitations. The Broader Impacts appeared to be addressed more
comprehensively towards the end of the 2000-2002 period, as the emphasis
on this guideline and the clarity of its definition increased within
NSF. Note that different reviewers often used different definitions of
"broader impacts," ranging from education, to societal, to applications
in other scientific disciplines. We are impressed that OPP's work to
define and communicate what "broader impacts" can entail has been
adopted NSF-wide. The resulting guidance should be very helpful, and its
effectiveness should be clear when the next COV review occurs in three
years.

***4.** **Do the individual reviews (either mail or panel) provide
sufficient information for the principal investigator(s) to understand
the basis for the reviewer's recommendation?***

Yes. The overwhelming majority of individual reviews (both mail and
panel) provide a considerable amount of specific, relevant, thoughtful,
and insightful feedback to justify the basis for the evaluation.

Most proposals were reviewed by more than three external reviewers. A
very small number of reviews were cursory and had very little
substantive information. The reviews are used effectively by program
officers to develop their recommendations whether to make an award, to
decline, or to request modifications to the scope and/or budget.
Communications to PIs regarding OPP actions were consistently clear, and
the reviews were routinely provided to principal investigator(s) along
with or shortly after the communication about NSF's decision on the
proposal. Thus, each PI was provided with sufficient information to
understand the reasons for NSF action on the proposal.

A few program managers went far beyond the requirements in terms of the
quality and substance of their communication to PIs, and other program
officials in OPP could benefit from having these program managers share
their approaches across the Office. Occasionally, a program manager
would request a PI to comment on a specific reviewer's questions prior
to advancing the recommendation for funding.

***5. Do the panel summaries provide sufficient information for the
principal investigator(s) to understand the basis for the panel
recommendation?***

Mostly. In most cases, the panel summaries provided sufficient
information for the PIs to understand the basis for the panel
recommendation. In cases where there was only a panel review, the amount
of information available to PIs was less than in cases where there were
multiple mail reviews, as well. Some summaries (especially a few done
early in the review period, before FastLane was available to support
panel documentation) were sparse in their details.

***6. Is the documentation for recommendation complete and does the
program officer provide sufficient information and justification for
her/his recommendation?***

Yes. The documentation for recommendations was complete, comprehensive,
and clear in all the programs the COV reviewed. Program officers in
Arctic Natural Sciences (ANS) and Antarctic Geology and Geophysics
(AG&G) did particularly thorough jobs summarizing the panel and/or mail
reviews and describing the rationale for the recommended
decision---including scientific merit, broader impacts, funding and
logistical constraints.

Recommendation: OPP should consider sharing examples of exemplary
write-ups by program officers across the Office, thereby helping
everyone continuously improve the quality, thoroughness, completeness,
and clarity of these documents.

**OPP Response:** OPP agrees in the concept of sharing "best practices"
for continuous improvement. We will take steps to expedite the sharing
of best practices.

***7*. *Is the time to decision appropriate*?**

The COV wholeheartedly supports the NSF goal of continuing to reduce the
time to decision. The dwell time for OPP proposals is somewhat longer
than the NSF average, but not significantly longer, considering the need
to integrate both logistical and scientific elements. Note that in the
case of negative decisions, it is especially desirable to have the time
to decision, including the communication of reviews to PIs, be at least
one month shorter than the interval between proposal deadlines, so that
the PI can revise and resubmit the proposal in time for the next
deadline. Dwell time for Antarctic proposals is longer than that for the
Arctic program, probably because of logistics. The COV is pleased that
between FY2000 and 2002 the dwell time was reduced substantially in some
OPP programs (but not in others).

[Recommendation: OPP should continue to expedite decisions, strive to
reduce dwell time, and notify PIs promptly. It is especially important
to provide reviewer comments to PIs on declined proposals, at least one
month before the next proposal deadline (typically
semi-annual).]{.underline}

## 

## 

**OPP Response:** Over the past year OPP has had a working review the
proposal process to determine where and how dwell time can be shortened.
The principal participants in the working group will discuss this issue
with all staff at the upcoming retreat.

# 

# 

# A.2. Implementation of NSF Merit Review Criteria

***8. Have the panel summary reviews addressed whether the proposal
contributes to both merit review criteria?***

Increasingly. All panel summaries address the intellectual merit and
quality of the proposals. For a sampling of proposals reviewed between
2000 and 2002, it appeared that approximately one-third of panel
summaries addressed "broader impacts," usually by describing potential
scientific impacts or significance beyond the immediate field. "Broader
impacts" was more consistently addressed in more recent panel reviews.
NSF's new requirement for "broader impacts" to be addressed in proposals
in FY 2003, and the very practical guidelines NSF has adopted (based
originally on the product of a working group OPP convened to follow up
on a recommendation of the 2000 COV) to help define what constitutes
"broader impacts," should make a difference in FY 2003 and beyond.

***9. Have the review analyses (Form 7s) addressed whether the proposal
contributes to both merit review criteria?***

Yes for Scientific Merit; Increasingly for Broader Impacts. All review
analyses addressed scientific merit and importance quite thoroughly.
Fewer than half of the review analyses addressed broader impacts. The
review analyses of recent proposals were much more consistent in
mentioning broader impacts than those prepared early in the period the
COV reviewed. Often review analyses focused the assessment of broader
impacts primarily on broader scientific significance, rather than
describing educational or societal impact, for example, as well. The
very practical NSF guidelines, mentioned in the COV response to Question
8, above, make clear NSF's broad definition of broader impacts, and can
be expected to be reflected in future review analyses.

# A.3. Selection of Reviewers

***10. Did the program make use of an adequate number of reviewers for a
balanced review?***

Yes. The number of reviews requested ranged up to 15, and the number
received ranged from 3 to 12.

***11. Did the program make use of reviewers having appropriate
expertise and/or qualifications?***

Yes. The breadth, depth and effort apparent in written assessment
indicate clearly that the program made appropriate use of reviewers with
expertise and relevance to the proposals. This is particularly apparent
in cases where reviewers identified and discussed such issues in
proposals as original and innovative science, sound logic, clear vision,
and scientific flaws. An additional indication of appropriate reviewer
selection is that reviewers provided comprehensive, constructive advice
to the PI.

***12. Did the program make appropriate use of reviewers to reflect
balance among characteristics such as geography, type of institution,
and underrepresented groups?***

Yes. The geographic distribution of reviewers for OPP proposals was
diverse, including reviewers distributed throughout the conterminous
United States, Alaska, and Hawaii, and sometimes from foreign countries.
Most of the reviewers were from universities, but many were employees of
colleges, government, industrial, or other non-educational institutions.
NSF did not provide the COV with information on the gender or ethnicity
of the reviewers. Many proposals included at least one woman reviewer;
there were some minority reviewers, based on gender/ethnic knowledge
possessed by COV members. OPP does not wish to disqualify volunteer
reviewers who choose not to disclose their gender/ethnicity.

***13. Did the program recognize and resolve conflicts of interest when
appropriate?***

Yes. All conflicts of interest appear to have been resolved. Many
jackets included correspondence from persons asking to be excused for
reasons of conflict of interest.

# 

# A.4. Resulting Portfolio of Awards

***14. Overall quality of the research and/or education projects
supported by the program***

Generally, projects receiving support were all highly rated by
reviewers. We take this as evidence that the research and education
projects being supported are of high quality.

***15. Are awards appropriate in size and duration for the scope of the
projects?***

Yes, in most cases. Generally, those projects that were selected for
funding received 100% of the support for the period requested, and in
some cases supplementary awards were also provided when requested.
Comparing OPP award size with NSF as a whole shows mean awards are
generally higher whereas median awards are generally lower. This may
reflect a somewhat higher number of large awards within OPP. Award
duration within OPP is similar to the Foundation as a whole.

In a few cases where project budgets were reduced at the request of NSF,
there was evidence that graduate student salaries (and possibly
tuition/fees) were removed from the budget. Graduate students are an
important contributor to NSF's "Outcome Goal for People." Furthermore,
if involvement of graduate students within a project is an integral part
of its educational 'scope' then the reduction or removal of this line
could be a negative, unless funding for the graduate student(s) was
provided by other sources. The COV found no evidence to indicate whether
the participation of graduate students was reduced or funded in other
ways.

[Recommendation: To provide specific measures of its contributions to
NSF's Outcome Goal for People, OPP should try to develop statistics on
the total number of undergraduate, graduate, and postdoctoral
researchers receiving support from OPP awards, and also on the number of
Masters and PhD degrees produced in the course of OPP-funded research
projects. It would be useful if these data could be sorted according to
specific programs in both Arctic and Antarctic science sections and
reduced to percentages in order to aid comparisons. The COV recognizes
that it will be difficult to collect quality data of this type, and
acknowledges that the community must be involved.]{.underline}

**OPP Response**: If the COV observation, that graduate student salaries
appear to be preferentially removed when proposals are not full funded,
is borne out after more detailed analysis, this would be very troubling.
There is some evidence that this occurs at the initiative of the PI. We
will discuss this issue with our Advisory Committee and other groups and
our Program Officers will be alert to this issue. We are strongly
committed to supporting education through our research awards.

Data currently available on number of undergraduate, graduate, and
postdoctoral researchers supported is based on budgets approved at the
time of award, and are shown below:

  ------------------------------------- ----------- ----------- -----------
                                        **2000**    **2001**    **2002**

  Post Doctoral Associates              89          125         118

  Graduate Students                     311         428         379

  Undergraduate Students                152         235         229
  ------------------------------------- ----------- ----------- -----------

Please note that these data may not represent how funds are ultimately
spent by a PI because of their flexibility to revise budget categories
subsequent to the award.

Recommendation: OPP should ascertain whether the exclusion or reduction
of budget line items for graduate student support has impacted the
number of young scientists supported and graduated to any significant
degree, and to follow up, as necessary.

**OPP Response**: OPP will initiate discussions within OPP and NSF
concerning ways to improve data on student support. We will consider
taking steps in post-award reviews to compare final outcomes with
initial plans, pending discussions at our upcoming staff retreat.

***16. Does the program portfolio have an appropriate balance of High
Risk proposals?***

Yes. It was clear from the jackets that scientific risk was not a
negative factor in proposal review and that presenting a scientifically
risky idea did not disadvantage proposals with respect to funding.
Logistical riskiness or complications occasionally resulted in favorably
reviewed proposals being delayed, modified, or declined. Some SGER
projects allowed rapid, preliminary exploration of high-risk concepts.

***17.* *Does the program portfolio have an appropriate balance of
multidisciplinary proposals?***

Yes. The program portfolio seems to be split rather evenly between
disciplinary and multidisciplinary proposals, and this is appropriate
for OPP. OPP by its very nature is a strongly multi-disciplinary
program.

***18. Does the program have an appropriate balance of innovative
proposals?***

Yes. Although reviewers are not asked to assess the innovativeness of
proposals, in about 10% of cases innovativeness was so significant that
reviewers noted it. There was no obvious discrimination against
proposals called 'innovative.' It would be useful to a COV, if the
reviewers (mail and panels) were asked for their opinion on this
subject.

***19*. *Does the program portfolio have an appropriate balance of
funding for centers, groups and awards to individuals?***

Yes. OPP appears to have an appropriate balance for the funding of
individual and group (taken to mean an interdisciplinary or
multidisciplinary project) awards. We saw no jackets that involved
'centers.' Both the funded and declined categories were almost evenly
split between individual and group efforts.

***20. Does the program portfolio have an appropriate balance of awards
to new investigators?***

Yes. A significant fraction of the awards go to new investigators, and
it appears that the reviewers and program officers are alert to
proposals by new investigators and somewhat tolerant of proposal
presentation weaknesses caused by PI inexperience. About 22% of OPP PIs
receiving awards between 2000 and 2002 were new investigators. OPP's
rates of funding new PIs (32%) is close to NSF's average (34%), as is
its funding rate for proposals from new PIs (OPP at 26% is slightly
above the NSF average of 23%). Thus, new PIs submit an average of 1.4
proposals to get an OPP award. This compares favorably with both the
NSF-wide rate of 1.6 for new PIs and the OPP overall rate for all PIs
(1.7). OPP should continue to be proactive to encourage new PIs and help
them learn how to develop successful proposals.

***21. Does the program portfolio have an appropriate balance of
geographical distribution of Principal Investigators?***

Yes. During the target period, OPP received and supported proposals from
a wide geographical area, although some areas were more highly
represented than others. Overall, the geographic representation is
broad, with grants having been awarded to institutions in 49 states
during the reviewed period.

***22. Does the program portfolio have an appropriate balance of
institutional types?***

Yes. For both Arctic and Antarctic programs the majority of successful
applications came from Doctoral Research Universities (DRU), primarily
DRU-Extensive. But the review process is not biased in favor of these
institutions. For the Arctic Program, 74% of awards and 81% of declines
are for DRU institutions. For the Antarctic Program, the portions are
81% of awards and 77% of declines. Master's Colleges and Universities I
have about 5% of the portfolio, and Liberal Arts Colleges have about 2%.
The remaining awards are to specialized institutions and "other."

[Recommendation: OPP should continue to be creative and proactive in
encouraging the participation of PUI (Predominantly Undergraduate
Institutions) in the research process, including encouraging partnering
between DRUE/I and other institutions and also use of RUI (Research in
Undergraduate Institutions) and REU (Research Experience for
Undergraduates) programs]{.underline}.

**OPP Response:** OPP will discuss ways to further encourage
participation of predominately undergraduate institutions and their
faculty at the upcoming retreat and with the OAC. See also the response
to Recommendation 25, below.

***23.* *Does the program portfolio have an appropriate balance of
projects that integrate research and education?***

Yes. Most funded proposals are for research, and they often provide
funding for graduate and undergraduate students training so they have an
educational aspect. We note that the program has also funded several
proposals that have strong educational (K-12) and community outreach
components.

***24*. *Does the program portfolio have an appropriate balance across
disciplines and subdisciplines of the activity and of emerging
opportunities?***

Yes. Examination of the proposals showed a good balance across
disciplines and sub-disciplines both in the Arctic and Antarctic
regions. Many proposals showed good potential for exploitation of
emerging opportunities in biocomplexity in the environment, information
technology research, nanoscale science and engineering, and workforce
needs in this century. The FY 2002 GPRA Performance Report and the OPP
Office Advisory Committee Reports on the GPRA were examined by the COV
and they support the above conclusions.

***25. Does the program portfolio have appropriate participation of
underrepresented groups?***

The COV determined that the OPP had rates of participation of
underrepresented groups that were close to or exceeded the rates for NSF
as a whole. Neither OPP nor NSF as a whole, nor most of the fields
funded by OPP has participation by PIs from underrepresented groups that
approaches national demographics. There is evidence that OPP pays
considerable attention to increasing the number of women and
underrepresented minorities among its awardees and in its communities.

[Recommendation: The COV encourages OPP to continue to be creative and
proactive in this area and attempt to further increase the proportion of
underrepresented groups.]{.underline}

**OPP Response**: OPP agrees with the recommendation. We intend to focus
our collaborative efforts with NSF's Office of Legislative and Public
Affairs reaching underrepresented groups, and will continue to support
workshops for new investigators, as well as efforts by individual
program managers to encourage broader participation. This will be a
topic for discussion at our upcoming retreat.

***26.* *Is the program relevant to national priorities, agency mission,
relevant fields and other customer needs? Include citations of relevant
external reports***

Yes. By nearly any conceivable measure, the NSF Office of Polar Programs
strongly and successfully supports the NSF Mission, and hence NSF's role
in national priorities. The Office itself exists in part to assure
continuous, coordinated support for the NSF Antarctic mission, and by
explicitly including Arctic science assures that high latitude science
issues receive focused attention within the overall NSF. Nearly every
funded project examined fit strongly and clearly into the Office of
Polar Program's role within the NSF. Many projects have explicit
educational ties, with all levels from elementary to graduate education
found in the sample. Workshops, databases, and web sites distribute and
exchange information and understanding nationally and internationally.
The "Augustine Report," the NSF/OPP GPRA Performance Reports for the
years 2000, 2001 and 2002, and numerous press clippings, provide
detailed documentation.

The special characteristics of the high latitude regions in terms of
response to climate change signals, as preservers of climate change
signals, as a laboratory to study unique and important processes related
to climate change, and physical and biological responses to climate
change are extraordinarily important to national and international
scientific priorities. Other unique aspects of Arctic and Antarctic
environments are represented in research related to anthropogenic
effects upon the environment, in social impacts of climate and land-use
change, in aesthetics via the Artist's program, in astrophysics and
aeronomy, and in opportunities for international collaboration.

The long-term continuity of some OPP observations fits well with
emerging priorities to obtain long-term observations in order to improve
understanding of processes and projections of future change.

## 

## 

## 

# A.5. Management of the Program Under Review

***27. Management of the program***

Management of the program is quite effective overall. The jackets were
well maintained, documentation was clear, the review process was run
well, and there is accountability, requiring annual and final reports.
Program managers use the flexibility allowed by NSF processes to good
advantage, and are able to tailor the approach to fit the needs of each
program. It will be important for NSF to continue to preserve this
flexibility. Some programs consistently provide decision documentation
that is exemplary. Reviews, both mail and panel, are used effectively by
program officers to develop their recommendation on whether to make an
award, to decline, or to request modifications to the scope and/or
budget. Communications to PIs regarding OPP actions were consistently
clear. A few program managers went far beyond the requirements in terms
of the quality and substance of their communication to PIs, and other
program officials in OPP could benefit from having these program
managers share their approaches across the Office.

However, the proposal pressure, success rate, dwell time, and their
trends vary significantly between the Antarctic program and the Arctic
program and among the subprograms. The demands on program officers vary
significantly, as well, with those responsible for Antarctic field
programs needing to spend extended periods in Antarctica providing
oversight. In its brief review, the COV developed some questions about
program management that it offers in the spirit of continuous
improvement, rather than criticism.

[Recommendation: OPP, perhaps assisted by its Advisory Committee (OAC),
should attempt to answer the following questions, and rectify imbalances
or unintended consequences, if they exist.]{.underline}

a)  Is each program appropriately staffed to balance the workload among
    programs?

b)  The Arctic and Antarctic programs are organized rather differently.
    This may be perfectly appropriate, but it raises the question of
    whether OPP overall is optimally structured?

c)  Given the number and scientific diversity of the proposals submitted
    to ANS, is its current organization and approach optimized?

d)  Would panels be feasible and valuable for providing a coherent
    ranking of proposals in ANS? Would mail reviews provide important
    additional information for panel use in ARCSS?

e)  The proposal success rate appears to be quite variable across OPP
    programs. Are these differences appropriate or should program
    allocations be flexible to respond to variations in proposal
    pressure and quality?

f)  In some cases, OPP requests that PIs adjust their proposals to fit
    within a budget below that requested. It appeared to the COV in its
    review of jackets, that budget reductions were disproportionately
    taken by reducing graduate student support. What is the impact of
    budget reductions on the research and on the inclusion of graduate
    students in the research?

g)  It appeared to the COV that proposals with only three mail reviews
    were more likely to be declined than proposals with larger numbers
    of reviews and/or consideration by a panel. Is this observation
    accurate?

[Recommendation: That OPP, assisted by its Advisory Committee (OAC),
develop answers, and, as appropriate, statistics on the above questions
during the next three-year period and provide these data to the next COV
committee.]{.underline}

**OPP Response:** OPP will develop data as suggested to address these
questions. The upcoming retreat will provide an excellent opportunity to
discuss these issues in depth with the entire staff prior to the next
OAC meeting.

***28.* *Responsiveness of the program to emerging research and
education trends***

The Antarctic Sciences and Arctic Sciences Programs of OPP invest in
research across an extremely broad range of disciplines, including both
specialized and multidisciplinary studies. There is no evidence that
program decisions are biased against emerging research trends and in
favor of traditional research projects and directions. Emerging research
trends tend to derive from new research directions, new discoveries, new
technologies, and opportunities presented from emerging or temporal
natural phenomena. The OPP programs are alert to and responsive to the
occurrence of time-dependent astrophysical, oceanographic, ecological,
climatologic, glaciological, etc natural phenomena. The SGER mechanism
is well used to take advantage of particularly timely opportunities. OPP
program officials are well networked with the disciplinary programs in
NSF and with the research communities doing polar research, thus they
are generally aware of emerging research trends. The Teachers
Experiencing Antarctica and the Arctic (TEA) program, which is coming to
an end, has been an excellent example of OPP's responsiveness to
emerging education trends and alignment with education best practice.
Through the TEA program, OPP provides valuable opportunities for K-12
teachers to participate in scientific research, learn first hand about
the nature of scientific inquiry, and transfer the experience to their
classrooms. In the new technology arena, OPP invests in a limited
portfolio of innovative and novel instrumentation concepts, which
promise significant benefits for future research. The Polar Instrument
and Development Program, which invests in these instrument concepts, is
extremely competitive, with only about one of five proposals able to be
funded. This represents an extremely high threshold for funding. The COV
suggests that OPP consider increasing the emphasis on new
instrumentation in its future portfolio, because of the leverage on
future research capabilities. In neither the jackets nor the
presentations did we learn whether or how much OPP research (versus
logistics) is taking advantage of emerging research trends and
capabilities in computation, computational science, and information
technology. This area may be one deserving of greater attention.

[Recommendation: OPP should find some way to enable teacher
participation in polar research to continue into the future, even as TEA
comes to an end. There should also be a mechanism found to continue
follow-up support of the existing cadre of TEA teachers.]{.underline}

##### **OPP Response**: OPP supports the goals of involving teachers in research in the polar regions and will continue collaborative efforts with EHR to encourage activities like TEA. A dear colleague letter was distributed last year specifically to encourage proposals in this area.

***29. Program planning and prioritization (internal and external) that
guided the development of the portfolio under review***

OPP's program is planned and prioritized in a variety of ways, which
include workshops sponsored by NSF, NSF/OPP, U.S. National Academy of
Science studies, international workshops, and leadership provided by
Groups of Specialists and Working Groups of SCAR (Scientific Committee
on Antarctic Research), for example. Furthermore, OPP can point to a
long and successful history in the promotion of activities that brought
or bring communities together to promote the planning, execution and
follow-up of innovative polar science. Examples of successful
initiatives from the Arctic include SHEBA (Surface Heat Budget of the
Arctic), and from the Antarctic include, WAIS (West Antarctic Ice Sheet
Project), CRP (Cape Roberts Project), and ANDRILL (Antarctic Drilling
Program).

These synergistic programs advanced science within and between major
disciplines and have guided the development of the portfolio. OPP also
participates in some of the large multidisciplinary NSF programs (JGOFS,
GLOBEC) and agency-wide initiatives, such as Biocomplexity, Nanoscience,
and the others. OPP should evaluate whether or not to promote the
initiation of more multidisciplinary programs.

# 

# 

# 

# 

# 

# 

# B.1 NSF Outcome Goal for People: Developing "a diverse internationally competitive and globally engaged workforce of scientists, engineers, and well-prepared citizens."

The COV concluded that OPP has performed at the highest level in the
strategic area of *Outcome Goal for People,* and that the investment has
resulted in many tangible and valuable results. We note the many and
varied people-related activities and results portrayed in the *GPRA FY
2002 Performance Report, pages 4-17.* The wide spectrum of program
activities in both the Arctic and Antarctic include the training of
undergraduate and graduate scientists in polar settings, via on-land and
at-sea field experiences, courses, internships, workshops and
conferences. OPP investments have contributed to the development of a
well-educated, very experienced and productive workforce of scientists
and engineers. The Teachers Experiencing Antarctica and the Arctic (TEA)
program has allowed K-12 teachers to bring the relevance of scientific
exploration and investigation and the scientific method into schools
across the nation in an exciting and innovative fashion. OPP has been
successful in its attempts to attract the participation of
underrepresented groups in the polar science enterprise. The
significance of science in general, and polar science in particular, to
a wide range of societal issues, has been clearly and successfully made
available to a large and interested public via documentary film, web
sites, radio, television, and the popular press, etc. The Antarctic
Artists and Writers program brings artists and writers to Antarctica,
and their art and writings stimulate and engage the large numbers of
people reached by their work.

# 

# B.2 NSF Outcome Goal for Ideas: Enabling "discovery across the frontier of science and engineering, connected to learning, innovation, and service to society."

The COV concluded that OPP has performed at the highest level in the
strategic area of *Outcome Goal for Ideas* and that the investment via
grant awards has been extremely well founded. We note the many
Idea-related results documented in the GPRA FY 2002 Performance Report,
pages 17-40, and also in the scientific publications that communicate
new ideas, discoveries and data to the scientific community, as well as
the popular science press which conveys this information to the public
at large. Scientific research in both the Arctic and Antarctic regions
spans the complete spectrum of the Geosphere, Atmosphere, Cryosphere,
Hydrosphere and Biosphere, and beyond into space. While all results
contribute to the essential polar knowledge base, we note the
significant contributions being made to a wide variety of global
research initiatives, and confirm the crucial role that polar science
plays in understanding planet Earth. A few examples of frontier science
and contributions to global issues include: deciphering the record of
Antarctic glacial cycles millions of years back in time and their
relationship to global sea level oscillations and global deep sea
climate cycles, implications for methane formation during climate change
in Alaska, discovery of the number and extent of large freshwater lakes
thousands of meters below the East Antarctic Ice sheet and the potential
impact of these sources of water on the global hydrosphere, archeology
revealed at sites associated with retreating Alaskan glaciers and
snowfields, and understanding of origins of North American populations
through DNA studies of human remains in the Aleutian Islands, etc.

# B.3 NSF Outcome Goal for Tools: Providing "broadly accessible, state-of-the-art and shared research and education tools."

The COV concluded that OPP has performed at the highest level in the
strategic area of *Outcome Goal for Tools* and that the investment was
both responsible and successfully applied. These outcomes are more fully
documented in the GPRA FY 2002 Performance Report, pages 40-49. The
scientific objectives of the Arctic and Antarctic research programs, and
the personnel involved in achieving these outcome objectives, will only
succeed if supported by modern and well-equipped facilities, efficient
and functioning equipment and instruments, state-of-the art
communications, adequate ground and air transport, and the highest level
of safety. OPP has continued to develop and modernize the logistic
support system for scientists and staff who work in both the Arctic and
Antarctic regions. Important items in connection with infrastructure,
logistics and environment include: building and refurbishing of bases
(including South Polar Station), logistic and service contractors,
aircraft for long-haul cargo and personnel transport operations,
fixed-wing and helicopter services for science parties, ships for
transport and use as science platforms, deep drilling rigs and camps for
use in support of glaciological and geological programs, repositories
for geological and glaciological materials, and environmental monitoring
and stewardship, etc.

# 

# C.1 Program areas in need of improvements or gaps within program areas

***1. Interdisciplinary, multidisciplinary, bipolar, and international
science ventures***

While the committee applauds OPP's extremely robust and long-running
record in the promotion of interdisciplinary, multidisciplinary and
international science, and leadership demonstrated by the United States
Antarctic Program over several decades, we believe there are compelling
reasons to continue augmenting such efforts, particularly with regard to
the preparations for the upcoming International Polar Year-4 program
(2007-08) that is to offer increased opportunities to do so.

[Recommendation: The Arctic and Antarctic science sections should
consider engaging with the community to conduct a series of 'future
workshops' or 'think tank meetings' which focus on the coupling of
bipolar science, the integration of polar-derived data with global
programs, the elucidation of new cutting edge opportunities between
disciplines, involvement with agency-wide initiatives, etc.]{.underline}

**OPP Response**: OPP encourages polar research to be well-connected to
global science and also encourages the community to consider \"polar\"
perspectives if appropriate rather than simply Arctic or Antarctic. Both
of the polar sections in OPP are open to ideas for workshops toward this
end. Because of the different roles that NSF/OPP plays in the Arctic and
Antarctic, it is equally important that workshops involve other agencies
where interests intersect.

### 

*2. Technology and logistics in support of science*

The COV focused on Arctic and Antarctic Science, Arctic Research Support
and Logistics (ARSL), and Major Research Instrumentation. It did not
review the Polar Research Support System (PRSS) for Antarctica. The
development of technology to support science is often done through ARSL
and PRSS. During the 2000-2002 period reviewed by the COV, OPP issued
one focused and highly oversubscribed solicitation for polar research
instrumentation development (Polar Instrumentation and Technology
Development Program). As polar science grows in scope and complexity the
need for improved research instrumentation will increase. This need
would be best met by offering more frequent announcements of opportunity
and/or requests for proposals for polar research instrumentation that
are coordinated by the Arctic and Antarctic Science Sections.

[Recommendation: OPP should seek adequate budgets to support the
development of new polar research instrumentation, link polar research
instrumentation development opportunities directly to research needs in
the Arctic and Antarctic, issue regular solicitations in this area, and
increase the number of awards for technological innovation and
development.]{.underline}

**OPP Response**: OPP was both surprised and pleased with the
community's response to the polar instrumentation development
solicitation several years ago. OPP would likely review the results of
the awards made under that solicitation before determining the nature of
any future instrumentation development activities. This will be
discussed at the upcoming retreat. We have now joined with the NSF
Engineering Directorate in the Sensors and Sensor Networks initiative
and are actively promoting increased attention by polar researchers in
the NSF MRI Program. Another good source of support is NSF's Directorate
of Computer and Information Science and Engineering Shared
Cyberinfrastructure initiative.

# 

# C.2 Program performance in meeting program-specific goals and objectives not covered by Core Questions A1-29 and B1-3

Not applicable or covered elsewhere in C1 to C5.

# C.3 Agency wide issues that should be addressed by NSF to improve the program's performance

***1. Agency-wide data bases***

Accurate and well organized numerical data are a critical element in
meaningful assessment of a program's performance over the three year
time frame encompassed by the COV review process. They are also
essential should an Office, Directorate or Division, or NSF at large,
mandate a survey which links a succession of COV reviews.

[Recommendation: The NSF should standardize its data collecting and
reporting procedures at all administrative levels and across all
Directorates, Divisions and Offices; ensure that non-sensitive data be
available on-line; provide complete indexing and cross-linking for these
data; and enable effective graphical formats for such data. Also, NSF
should ensure that it collects and makes readily available to the COVs
in advance whatever data are needed to answer the questions it sets for
COV reviews of Directorates and Offices.]{.underline}

**OPP Response**: For the 2003 COV, OPP developed considerable data and
made them available, but some data were simply not obtainable. A
Business Analysis currently being conducted by an outside contractor is
reviewing NSF's various IT systems and how they might be improved,
including the issue of cross-linking data. OPP agrees that NSF needs to
give more thought to ensuring that data necessary for addressing
questions posed to COV's is available. We will discuss this issue the
those in the agency who prepare the COV template.

# 

# C.4 Other relevant issues

***1. Intellectual property rights with regard to indigenous people and
local communities***

This issue centers on problems arising when researchers study the
long-held cultural traditions, a specific type of expertise, and/or
environmental knowledge held by indigenous people or by particular local
groups. Whereas the academic/research community and general public may
benefit by getting access to such knowledge, we pose the question as to
the benefits that are accrued by the communities in which such
expertise, knowledge, and traditions originate. The OPP/COV suggests
that NSF ensure that interactions with Arctic communities be conducted
in ways that facilitate future research; and seek information from
institutions and groups (such as the Alaska Native Knowledge Network,
Nunavut Research Institution, American Association for the Advancement
of Science, etc) on their approach to respecting and dealing with this
type of intellectual property rights.

[Recommendation: The relevant programs within NSF should examine the
issue in depth and develop policies and guidelines for dealing with
intellectual property rights associated with indigenous people and local
communities.]{.underline}

**OPP Response**: The U.S. federal government is currently participating
in international negotiations on the global issues surrounding the
intellectual property rights of indigenous people. The U.S. Patent and
Trade Office and the Department of State are participating in a number
of international fora discussion how to define traditional knowledge and
what intellectual property rights (IPR) should be afforded to the use of
indigenous knowledge. These include the Convention on Biological
Diversity (CBD) http://www.biodiv.org/default.aspx, the World
Intellectual Property Organization (WIPO) http://www.wipo.org/, The
World Summit on the Information Society (WSIS) http://www.itu.int/wsis/.
NSF is participating in the interagency process but is not the lead US
agency in these international fora. Therefore, until the U.S. makes an
official determination of what is in its best interest regarding these
issues it would be inappropriate for NSF to take independent action by
developing internal guidelines and policies. For example, the Department
of State recently chaired an interagency group that developed guidelines
for U.S. researchers on how to comply with the Convention on Biological
Diversity when collecting specimens in countries party to the
Convention. NSF recently posted a link to the guidelines on its website
(<http://www.nsf.gov/home/int/travel.htm>); the direct link is
http://www.state.gov/g/oes/rls/or/25962.htm.

In the Arctic, NSF funded researchers have a responsibility to implement
the Principles for Conduct of Research in the Arctic
(http://www.nsf.gov/od/opp/arctic/conduct.htm). These principles include
many of the issues surrounding the discussion of intellectual property
rights such as: informing the community and obtaining consent; respect
for local cultural traditions, languages, and values; communicating with
communities on a continuing basis; and providing meaningful experience
and training for local people.

***2. Strengthening assessment of and accountability for Broader
Impacts***

OPP programs send proposals primarily to scientific/engineering peers of
the principal investigator, and request their evaluation of the proposal
with respect to its intellectual merit and quality and broader impacts.
The requirement for researchers to include broader impacts in their
proposals helps get both investigators and reviewers to think about them
and to be proactive in achieving them. However, these individuals may
not be qualified to develop or assess certain aspects of broader
impacts, without getting input from individuals expert in a
broader-impact area, such as education, policy, international relations,
economics, or other fields. Such collaboration can be very useful in the
development of a proposal, in the review of a proposal, and in the
conduct of the work. Over the past few years, NSF has continuously
strengthened and clarified its expectations with respect to broader
impacts. Most recently it implemented a requirement in FY2003 that
proposals not addressing broader impacts be returned to the PI without
review. In a similar way, NSF could phase in an expectation that the
PI's and research team's previous accomplishments in "broader impacts"
be mentioned in the proposal section that summarizes the results of
prior work, so that reviewers can judge the PI's and team's track record
in this arena, as well as on their scientific track record.

[Recommendation: OPP should consider requiring PIs to summarize their
track record in "broader impacts," in the section of the proposal
describing prior results. In addition, where it would be beneficial, OPP
should seek to ensure that the mail or panel reviewers include
individuals with strong qualifications to assess and provide feedback on
the broader impacts of a proposal.]{.underline}

**OPP Response**: At our staff retreat we will discuss how OPP's Program
Officers can facilitate this. In addition, and as this is an NSF-wide
issue, OPP will initiate discussions within NSF about this
recommendation.

***3. Ensuring that information essential for accountability is obtained
from the most appropriate source at the most appropriate time***

The COV plays an important role in NSF's process of assuring
accountability for getting high value from its taxpayer funds. Yet the
COV is not the only mechanism available for assuring accountability, and
many of the assessments the COV is requested to make cannot be made
effectively during the 3-day COV review. Such assessments include
judging the innovativeness, riskiness, and other aspects of the
portfolio, when to do so, etc. To achieve this we would have to read and
judge the innovativeness, riskiness, etc of each of nearly 200 proposals
comprising the random sample of jackets. At the risk of adding more
check boxes (and other bureaucracy) to the review process, we make the
following recommendation.

[Recommendation: NSF should make final reports from grants available to
the public through its Web page. In addition, NSF should study its "core
questions for COVs" and determine how to acquire the best information on
each question and make this information available for COV use. For
example, perhaps each proposal reviewer (mail and panel) should be asked
to assess and indicate the innovativeness, riskiness,
multidisciplinarity of each proposal, and this information could be
captured in the statistical data maintained by the agency.]{.underline}

**OPP Response**: The OPP Director has raised the question of posting
final reports on the web with agency management in the past and will do
so again. We agree that they should be publicly posted. See also the
response to the COV recommendation in C.4.2.

# 

# C.5 Improving the COV review process, format and report template

***1. The COV process overall***

The COV consists of a group of scientists and engineers with expertise
generally spanning the disciplines and research areas covered by OPP
programs, who spend three days at NSF reviewing the management, proposal
"jackets," and outcomes of OPP's investments. The work of the COV is
guided by "NSF FY2003 Core Questions for COVs." Most of these questions
each focus closely on one specific aspect of the process for reviewing
and investing in proposals and the demographics of people engaged in the
program. Some of the questions request an assessment of matters the COV
is well qualified to judge. Others request assessments, for which the
COV possesses no special qualifications. Finally, it is easy given the
large number of very specific questions, the requirement to address each
one for NSF's GPRA accountability, and the small amount of time for the
COV to "miss the forest for the leaves."

[Recommendation: NSF should reduce the number of specific questions it
asks of COVs, and emphasize those that are most important to its
performance and accountability that are of a type appropriate for COV
assessment. In addition, OPP should identify a few areas in which the
COV could provide advice and assessment that would be most important and
helpful to OPP's management and outcomes, and pose questions/issues in
those areas for the COV consideration, along with the standard NSF
direction to COVs.]{.underline}

**OPP Response**: OPP agrees with this recommendation. While the COV
standard template provides the flexibility to tailor questions to a
particular organization, NSF has been criticized because not every COV
has addressed every question. Striking the right balance between getting
information that improves management oversight, on the one hand, and
performing exhaustive audits, on the other, has proved difficult for
NSF.

***2. Agency-wide software-based COV template***

The review of information and data, and writing of a comprehensive
report within the span of a three day meeting is no simple task. The
development of a standard agency-wide COV software package, such as is
used by NSF panels, and in which the template categories or headings are
set out in the format of a final report, might make the process more
efficient.

***3. Technical facilities***

Ideally, the COV report is prepared in three days. For the FY 2003
review OPP provided net-worked laptop computers for all committee
members. While we made ample use of the computers we did not have time
to make full and efficient use of these facilities. Partly due to the
intervention of 'Hurricane Isabel' we ended up preparing text and
transferring this via floppy disks to a master template on a separate
laptop.

[Recommendation: The next COV chairperson should work with OPP staff to
prepare a final report template based on the Core Questions in advance
of the meeting, and have this template pre-loaded onto the laptop
network before the meeting. It would also be useful to have appendices
entered into the template ahead of the meeting.]{.underline}

**OPP Response**: This is an excellent recommendation that we will seek
to have implemented for future COVs.

***4. Assembly, availability, review and presentation of the OPP data***

The 2003 COV examined two large sets of material. The first set consists
of the randomly selected proposal jackets representing awarded and
declined proposals, in all the major disciplines within the Arctic and
Antarctic science sections. This material is strictly protected by
confidentiality law and could not have been examined before the COV
meeting. The second category of material includes large amounts of
statistical information and reports, which are publicly available.

[Recommendation: OPP should provide the COV approximately 8 weeks in
advance of the meeting with the list of solicitations, a bulleted
summary of the evaluation criteria for each solicitation, along the
materials routinely sent in advance to COVs and all of the publicly
available statistical information related to NSF's core questions for
COVs. The COV chairperson should consider delegating reviewing tasks to
individual committee members at that time.]{.underline}

**OPP Response**: OPP agrees with both of these recommendations and will
make materials available in advance on a special web page, as well as
ensuring future COV chairs have seen all of these recommendation,
including the advice to assign tasks in advance of the meeting

[Recommendation: OPP should prepare a list of 'non-sensitive\' data
topics that are deemed central to the COV process, compile and present
data and source information for each topic, and organize these as a
series of numbered appendices for the COV report.]{.underline}

**OPP Response:** OPP agrees with this recommendation. While this was
provided at the meeting to the COV members, consistent with other
recommendations, it will be provided in advance with minor exception
(e.g., detailed information on declines, which is considered
privileged).

[Recommendation: OPP should collect and maintain data on the total
number and diversity of undergraduate, graduate, and postdoctoral
researchers receiving support from OPP awards, and also on the number of
Master's and PhD degrees produced in the course of OPP-funded research
projects each year. The COV recognizes that it will be difficult to
obtain such data, and that the community will need to provide it. It
would be useful if these data were able to be sorted according to
specific programs in both Arctic and Antarctic science sections, reduced
to percentages to aid comparisons, and provided to the COV in advance of
the review.]{.underline}(See A.4.15)

***5. The long term view***

Although the FY2003 COV considered aspects of the 2000-2002 time frame
in considerable detail, we did not concern ourselves with pre-2000
records. This was not included in our charge and would probably be made
difficult because NSF Core Questions tend to change and evolve over
time.

[Recommendation: NSF should consider the strategic value of
standardizing some aspects of basic data collection and the COV
assessment, so that the next COV could evaluate selected long term
trends within OPP. In fact, assessing trends might be more meaningful
than the absolute judgment of "appropriateness," or "appropriate level"
required for some of the core questions.]{.underline}

**OPP Response**: OPP agrees that long-term trends are often meaningful
to program review and assessment and will make longer-term data
available to future COVs.

***6. Devising the COV work plan***

In choosing the most effective and efficient work plan we considered;
(1) having each member address all core questions, (2) separating
proposals into Arctic and Antarctic groups before review, (3) separating
proposals into science program specialties (e.g. marine and terrestrial
biology) with Arctic and Antarctic proposals combined, and (4) assigning
a limited number of core questions to a pair of committee members who
would then examine all proposals without geographic or discipline
distinctions being made. We opted for (4). This allowed for a much more
in-depth assessment, the tallying of quantitative data, more objective
detection of patterns and trends, and very lively presentation of the
assessment with respect to each question to the entire COV during the
wrap up and writing phases of the COV meeting.

***7. Committee of Visitors and Office of Polar Program staff
interaction***

Despite the chaos wrought by Hurricane Isabel during the meeting, the
FY2003 COV process was a cheerful, positive and very productive
experience. The COV found great value in the well illustrated
presentations by Drs. Karl Erb, Thomas Pyle, and Scott Borg. It was also
helpful to meet OPP program managers during this introductory session.
This part of the meeting provided useful syntheses on OPP history,
staffing, budget and performance data, past and planned initiatives, and
interactions with other parts of NSF. These presentations and the
opportunity to pose questions allowed the committee to become acquainted
with the task ahead, and also become acquainted with each other. We also
note that Dr. Erb and staff were most helpful in answering questions and
providing additional information as the meeting progressed. Ms. Brenda
Williams very effectively provided administrative and logistical support
for the COV, which allowed its work to be accomplished despite the
hurricane