Update to the 2002 Committee of Visitors (COV) Report

NSF Division of Ocean Sciences (OCE)

Integrative Programs Section (IPS)

**As noted below, COV Findings/Recommendations, 2002 NSF Response, 2004
NSF Update:**

**Findings/Management Practices:**

-   *Utilization of Cooperative Agreements.* The Ship Operations program
    changed to five-year cooperative agreements in FY 2000 as the
    instrument for funding ship operations. The new five-year continuing
    Cooperative Agreements provide NSF with more flexibility to adjust
    annual budgets as cruise schedules change and as unanticipated needs
    arise. Program officials might consider applying Cooperative
    Agreements to other activities, such as the University of Miami SWAB
    program, that are considered routine undertakings.

**2002 NSF Response:**

Cooperative agreements were established by NSF, in recognition that
support of activities other than research required a higher level of
continuing interaction between NSF and the grantee, compared to research
grants. As the COV noted, in FY 2000 we transferred the Ship Operations
and University-National Oceanographic Laboratory System (UNOLS) Office
awards from grants to cooperative agreements. We have gained experience
with them, and it is now time, as the COV suggests, to examine other
similar activities, such as a long-standing grant to the University of
Miami to monitor and clean up shipboard radioactive contamination, to
determine whether they are more appropriate as cooperative agreements.

**2004 NSF Update:**

Use of Cooperative Agreements continue in IPS as the ability to
negotiate and provide better oversight lend itself to many of the
Section endeavors. Recent mid-size infrastructure awards to Lamont
Doherty Earth Observatory for the replacement of the R/V Ewing and to
Woods Hole Oceanographic Institution for the replacement of the Deep
Submergence Vehicle (DSV) Alvin were made as Cooperative Agreements. A
Cooperative Agreement was recently made to Joint Oceanographic
Institutions to manage the ORION Office. We envision that future
projects such as the construction and operation of an Alaska Region
Research Vessel (ARRV) and three Regional Class vessels will also be
made through Cooperative Agreements. The COV referred to "routine
undertakings" as a criterion for a cooperative agreement, but this
mechanism is most appropriate for complex undertakings which require a
great deal of ongoing NSF management. The COV seems to assume
(incorrectly) that a grant should be solely for original research.
Although the example they cite is not research, it is routine, and
requires essentially minimal ongoing NSF management.

-   *Coordination of fleet activities.* OCE is to be commended for its
    insight and leadership in use of group purchases of shipboard
    equipment for cost saving and standardization across the fleet. For
    example, bulk purchases of immersion suits by LDEO, work boats
    through LUMCON, and INMARSAT units compatible with SEANET through
    UDEL, were distributed across the fleet to those who had requested
    and received approval from OCE. Of particular note, is the benefit
    that resulted from this group purchase approach for laboratory vans.
    UDEL took the lead in designing standardized vans consistent with
    U.S. Coast Guard criteria for construction of sea-going vans
    occupied by scientists. A group purchase of several types of these
    vans provided standardized equipment to the fleet at considerable
    cost savings. Recommendations below suggest that this management
    practice might be applied in other areas as well.

Other best management practices include sharing examples of well written
program summaries with submitters of other proposals, the high level of
interaction between program officers, PIs, and other related NSF program
managers, and the practice of funding proposals based upon estimated
costs at the beginning of the year in order to assure funds are
available to keep the fleet operating without interruption.

**2002 NSF Response:**

The COV commended OCE "for its insight and leadership in use of group
purchases of shipboard equipment for cost saving and standardization
across the fleet", and recommended that we expand this management
practice, as outlined under the following two headings (Safety Training
and Security). Besides the bulk purchases of immersion suits, work
boats, communications electronics and shipboard laboratory vans, we have
long supported a fleet-wide winch and wire pool. We have every intention
of continuing to expand group purchases into other areas.

**2004 NSF Update:**

IPS continues to do bulk purchases when appropriate.

-   *Safety Training.* We applaud the NSF initiative and dedication to
    strengthening the overall level of training among the UNOLS ship
    crews. The commitment in 1999 to provide supplemental funding
    "across the board" in support of the Seafarer's Training,
    Certification and Watchkeeping Code -- STCW95 -- per guidance from
    the International Maritime Organization -- is characteristic of the
    community leadership exemplified by NSF. This investment, while
    hardly insignificant (several hundred thousand dollars), will pay
    off in terms of ensured safety of operations.

Review of the individual institutional agreements and budgets reveals a
wide diversity of costs associated with equivalent training packages.
For example, costs for STCW-BST (Basic Safety Training) ranged from
\$875 per individual, to more than \$3000 per individual in the
proposals that we examined. Similar variability in cost was noted for
other courses as well. We recognize that part of this range in costs is
attributable to differences in services provided by the different
instructional facilities around the United States (e.g. some include
housing costs while others do not). We also recognize that in 2000 and
2001 the initial training initiative focussed on a larger-than-normal
population of the ships' crews (providing training for about 150
individuals). Nevertheless, NSF's clearly stated, long-term commitment
to providing a steady stream of resources for STCW training suggests the
need for a community-based approach to this issue. Lessons learned from
the community-wide approach to pooled equipment and group-buys of
equipment might be applied to training. We recognize that the logistics
of training crews, when taking into consideration cruise schedules and
geographic constraints, might be too daunting to address in any
centralized fashion. However, given the four-fold range in costs for
training, NSF is encouraged to assess opportunities for gaining
efficiencies (in costs and scheduling) though a centralized management
or tracking of training.

**NSF Response:**

The COV "applauded the NSF initiative and dedication to strengthening
the overall level of training among the UNOLS ship crews". The COV
recommended a community-based, rather than individual operator approach
to this issue, noting a four-fold range in training costs requested by
individual ship operators. NSF is encouraged to assess opportunities for
gaining efficiencies (in costs and scheduling) through a centralized
management of tracking and training. We agree that we should look to the
establishment of such a fleet-wide system.

**2004 NSF Update:**

Through the Ship Inspection Program, NSF is working with UNOLS and
OCEANIC (University of Delaware) to develop a database that will contain
a training inventory component. This component should help identify and
track crew training. It is hoped that through tracking training, we will
be able to help assess outstanding training needs throughout the fleet,
potentially arrange for group training experiences to save on costs and
provide training uniformity across the fleet. In FY 2003 we specifically
solicited requests for training of all kinds and spent \$250K in
addition to the normal annual funds, which totaled \$500K in FY2004.
Additionally, the SSSE program set up a winch maintenance training
program conducted on site at the ship by the winch manufacturer.

-   *Security.* Many ships in the UNOLS fleet routinely operate in
    international waters, and a few routinely operate in regions that
    have become increasingly prone to piracy and acts of international
    terrorism. U.S. oceanographic research vessels may become prime
    targets for such acts. Several weeks prior to September 11, 2001,
    one vessel operating in Somalian coastal waters was fired upon and
    actively pursued by pirates or terrorists. Fortunately, no bodily
    harm to the scientific party and crew was sustained. At least two
    institutions have proactively engaged in security training. Given
    the current international climate, the need for security awareness
    training for all vessels including those operating only within U.S.
    waters will most likely increase. To date recommendations and
    practices for security training, and more extensive security
    practices for vessels operating in unfriendly waters, do not appear
    to exist.

**NSF Response:**

The COV noted a recent attempted piracy attack on an NSF-owned ship (R/V
Ewing) in the Red Sea and the fact that recommendations and practices
for security training do not appear to exist. Security training across
the fleet had been funded by NSF a decade ago when a similar incident
occurred. Both the Ewing and another vessel operating in the Red Sea
region in the same year had undertaken security training and
precautions, but the COV is correct in that there are no systematic
ongoing procedures. Soon after the recent incident, UNOLS held a
workshop in Washington D.C. to seek advice of experts, and formed a
Security Subcommittee to examine all aspects of the issue and make
recommendations. As soon as the report is received, we will study its
recommendations carefully and develop an integrated training plan. In
the meantime, we maintain contact with Mr. Charles Dragonette of the
Naval Intelligence Office, who publishes the weekly Worldwide Threat to
Shipping newsletter, and who attends our Ship Scheduling meetings and
reviews our scheduling requests, advising us on any potential problems.
We

have also encouraged ship operators to request funds for security
training in their upcoming round of annual proposals.

**2004 NSF Update:**

The tracking and funding of security training will be done through a new
database system mentioned in the response to Safety Training above.
Several operators that navigate in international waters have taken
advantage of security training opportunities. We have avoided entering
"unfriendly" waters and ports (as determined by Department of State) and
continue to consult with Mr. Charles Dragonette during ship scheduling
meetings and on specific issues when needed.

-   *Ship Inspections.* OCE supports ship inspections by an independent
    contractor. These inspections provide operators with independent
    assessments of the state of their vessels, their capabilities and
    their potential for improvement. The COV noted several instances of
    positive references to the inspection process in the review of
    proposal jackets. The inspection program underwent a hiatus for
    several years while the proposal was changed from a grant to a
    contract. The program is of sufficient importance that such lapses
    should be discouraged.

**NSF Response:**

### The Committee noted that the Ship Inspection program underwent a hiatus for over two years, when the mechanism of funding was changed from a grant to a contract. They recommended that such lapses should be discouraged. We very strongly agree with this statement. The contracting process proved to be much more time consuming than we had anticipated, involving drawing up a highly detailed bid package, going out to bid, establishing a review process, a subsequent call for final bid offers and review. We will do everything we can to avoid another such lengthy hiatus now that we are familiar with the very different and complex rules of procedure compared with a grant. 

### 

### 2004 NSF Update: 

The current ship inspection contract will be completed the end of June
2005. Efforts are already underway to prepare a new contract for bid and
ensure that it will take effect with no hiatus.

-   *Merit Review.* Each program has adapted a flexible proposal merit
    review process that is appropriately scaled to their needs. In
    general, as described in proposal jackets, intrinsic intellectual
    merit is not the primary review criterion for facilities. Instead,
    the intellectual merit review occurs in a separate review process of
    the research proposals that use these facilities. Proposals
    submitted to the appropriate disciplinary science programs or
    interdisciplinary special solicitations are reviewed by mechanisms
    appropriate to each program or solicitation for both their
    intellectual merit and their appropriateness in using the
    facilities. Although it was inappropriate to comment on the
    \"intellectual merit\" of the science awards for which the
    facilities were provided, the Committee carefully examined the
    \"technical merit\" of the facilities proposals from the standpoint
    of mail, panel and review analysis. Additionally, the Facility
    Program reviews focus extensively on Criterion II of the guidelines.
    However, it is clear from examining the program jackets that when
    intellectual merit review of a broader nature is appropriate (such
    as the NOSAMS renewal proposal), that such review is instituted.
    These reviews often consist of large numbers of mail reviews by a
    diverse set of reviewers, followed by a special panel.

**Recommendations:**

-   *Quantitative Assessment of Service Quality.* The Ocean Science
    Facilities Programs manage a complex set of facilities equipment and
    science support infrastructure in support of ocean observation
    projects. The overall success of these efforts is measured by the
    scientific results reported in reviewed publications. However, these
    metrics do not provide useful direct links to Facilities Programs
    for assessing quality of service and evaluation of continuous
    improvement in support of the underlying science mission. The
    program managers very professionally compensate for this by their
    demonstrated detailed knowledge of the facilities, operations and
    scientist experiences.

However, development and use of high-integrity measures of service
quality would provide a robust, defensible means to help panels and
managers identify opportunities for improvement and a gauge for outcomes
of instituted changes (e.g., recent efforts in Technical Services
Program to fund a more uniform set of shared-use equipment to reduce
effects of ship reassignments to individual science PI budgets).

Post-cruise UNOLS assessments, which provide ample opportunity for
subjective interpretation, may not be appropriate for rigorous
analytical use. It may be necessary for NSF to develop more rigorous
measures of efficiency and quality of service.

**NSF Response:**

The COV noted the subjective nature of post-cruise reports and the need
for measures of service quality to facilitate evaluations, leading to
quantitative improvements. The old UNOLS post-cruise assessment forms
were indeed qualitative and subjective, and their revision has been the
topic of UNOLS Council discussion for over two years. A new form was
approved and introduced about the time of the COV meeting. The new form
is quantitative (using a variation of the standard NSF 5-point rating
system) as well as narrative, and solicits input from multiple sources
(chief scientist, captain, ship operator, and, in fact, any other
interested party who wishes to provide input). We have received the
first few submissions of the revised assessment form and so far they
appear to provide much more useful information. We will analyze the
results of the first year and make recommendations to UNOLS for any
improvements necessary to the form, or as the COV suggests, introduce
our own mandatory system of reporting. Further, Dr. Linda Goad who
became Program Director for Ship Operations in FY 2001, will apply her
Six Sigma training to rigorously pursue issues of quality assurance in
the Fleet. Under Dr. Goad's direction, a workshop of experts was
convened at NSF in September to explore in depth the continued failures
associated with Z-Drive propulsion systems, and identify the root causes
and the long-term corrective measures.

**2004 NSF Update:**

Service quality utilizing the new Post Cruise Assessments is underway.
Results of the user assessments on a 5-point scale for each ship and
each assessment attribute have been tabulated. Assessment results will
be provided to reviewers for evaluation during the upcoming (December
2004) external merit review of Ship Operation proposals.

-   *Availability of Ship Inspection Recommendations and Follow-up.* Web
    based tracking of compliance with recommendations of UNOLS ship
    inspections should be considered. Operators must detail, in each
    annual proposal, the recommendations of the previous ship inspection
    and the modifications and repairs that are being made in response to
    the recommendations. Such lists and the responses could be
    maintained on a web site (if privacy issues can be resolved) that
    would be available to a broad set of stakeholders, including the
    users of the ships. Awareness of the recommendations and responses
    would encourage greater input from users regarding perceived
    deficiencies in facilities, it would encourage continuous
    improvements of the facilities, and it would promote awareness of
    the efforts made by operators to accommodate science users.

NSF Response:

The COV recommended that web-based tracking of compliance of UNOLS ship
inspections should be considered if privacy issues can be resolved. A
new contract for ship inspections was negotiated by the NSF Division of
Acquisition and Cost Support and implemented this year. With its
establishment, we require ship operators to provide an initial response
to inspections within 60 days. In addition, we require the ship operator
to address the progress of any modifications/repairs, or request funds
to complete these where necessary, in the following annual ship
operation proposal. We will implement this recommendation.

**2004 NSF Update:**

As mentioned in the Safety Training section above, NSF is working with
UNOLS and OCEANIC (University of Delaware) to develop a web based,
password protected ship inspection reporting system. The system will
contain 3 major components, with potential for future expansion as
needed. The first major system component, the ship condition form, which
provides detailed information about the ship condition, personnel and
training, and ship equipment/ instrumentation inventory, is nearing
completion. The second and third system components will encompass the
NSF inspection team assessment of the ship condition and recommendations
and the ship operators response to the inspection recommendations.

-   *Centralized Tracking of Safety Training.* We recommend that NSF
    support the development and maintenance of a centralized, web-based
    tool for tracking STCW training throughout the community of UNOLS
    ships' crews. This tool, if accessible by the UNOLS community, will
    facilitate 'comparison shopping' for training opportunities. With
    appropriate links to training service providers, this tool will
    offer UNOLS marine operators a more efficient and effective means
    for attaining the best training programs, at the best prices to fit
    their schedules. Operator-maintained databases of crew training
    status, if resident on this site could also present the NSF program
    managers the single-site reference for planning future training
    resources.

NSF Response:

Similarly, the COV recommended that we support the development and
maintenance of a centralized, web-based tool for tracking personnel for
the Seafarer's Training, Certification and Watchkeeping Code training,
which could facilitate "comparison shopping" for the best and most
cost-efficient training programs. This is a good idea, but we believe it
should be implemented by UNOLS, because the official NSF site cannot be
used for such implied endorsements of commercial enterprises. We will
ask UNOLS to implement this recommendation.

**2004 NSF Update:**

As mentioned in the Safety Training section above, NSF is working with
UNOLS and OCEANIC (University of Delaware) to develop a web based,
password protected ship inspection reporting system that will
incorporate a component that can track training.

-   *Clarification of Guidelines for Shipboard Facilities and Operating
    Proposals.* Extensive experience with fleet operations has been
    accumulated. In some cases, additional guidelines for proposals may
    now help clarify allowable costs and help define reasonable
    expectations for the level of operational support needed to sustain
    ship operations in normal circumstances. In particular, there
    appears to be some confusion regarding guidelines for submission of
    proposals for instrumentation and equipment, in spite of NSF's good
    efforts to clarify the program constraints. Effectively, each
    institution has two different opportunities for proposing purchase
    of equipment and/or instrumentation defined by NSF as items costing
    over \$5,000 (supplies such as personal computers, can also be
    purchased under ship operations Cooperative Agreements): i as part
    of the Shipboard Scientific Support Equipment program, or; ii. as a
    separate Oceanographic Instrumentation proposal. For the most part
    there is consistency within and between these programs regarding
    what is and is not allowable. However, we noted several instances of
    requests being rejected from one program as inappropriate, and being
    redirected to another program. Comments such as *"The panel felt
    that shipboard computers should be replaced as they become obsolete
    through ship operations"* appear somewhat arbitrary and reflective
    of a need for even clearer guidance. Recognizing that there will
    always be some confusion in this regard, the number of such
    "redirections" is reflective of a misunderstanding across the
    community about the guidelines for submission of
    equipment/instrumentation requests. NSF needs to provide a clearer
    definition of the relevant criteria and delimiting factors for
    submission of such requests to each of the three relevant programs.

In reviewing Ship Operations proposals, it was also clear that there
were large variations in expectations for allowable levels of support,
both shipboard and on shore, that were required to sustain marine
operations. The variation in expectations appeared to produce some
contention. Clearer guidelines would allow operators to identify areas
where additional documentation would be required to justify enhanced
support levels.

NSF Response:

The Committee believes that NSF needs to provide a clearer definition of
the relevant criteria and delimiting factors for submission of proposals
to different Facilities programs. We will strive to reduce any apparent
confusion about which of the Programs is responsible for specific
equipment/instrumentation requests in a formal revision of the Program
Guidelines, which we hope to have published before the end of this
calendar year. The guidelines were last published immediately prior to a
Division reorganization three years ago, and do not include reference to
either the Ocean Technology Program (for technology development), which
is now part of this Section, or the Major Research Instrumentation (MRI)
Program, a centrally funded NSF program for which this Section has
administrative responsibility within the Division of Ocean Sciences. We
will try to devise a useful roadmap, which directs a PI to the
appropriate program for submission of equipment/instrumentation
requests.

**2004 NSF Update:**

The Guidelines to which the COV referred have been very extensively
revised and modified over the past two years. They were recently cleared
through the NSF clearance system and are now posted on the NSF web site.
This revision was in direct response to this COV recommendation.

-   *Documentation of Program Decisions.* Review of the proposal jackets
    found that program decisions were generally well documented. When
    questions arose regarding decision making, the missing component was
    usually identified as information that was passed on to proposers or
    awardees during an informal contact. Diary notations of such
    contacts would greatly increase the "readability" of program
    jackets. The previous COV (1997) made a similar comment.

NSF Response:

The COV noted that program decisions (with regard to ship operations)
were generally well documented, and recognized that contacts between the
Program and ship operators often were on an informal basis. In fact,
because of the dynamic nature of ship operations, ongoing contacts can
sometimes be on a weekly, if not daily, basis between the Program, the
operator, other interested agencies, PIs and science Program Officers.
We recognize that sometimes this "fog of operations" may be difficult
for an external observer (such as a COV member) to understand from the
written notations of already bulging Cooperative Agreement jackets. The
COV recommends that, "diary notations of such contacts would greatly
increase the "readability" of program jackets." We will try to find a
practical way to achieve this goal.

**2004 NSF Update:**

Every effort has been made on the part of the Ship Operations program
Officer to provide clearer documentation in jackets of continually
changing events.

**Future Emphases:**

-   *Program Structure.* The COV recognizes and commends the Ocean
    Science Facilities programs for their community-oriented perspective
    in prioritizing platforms, support and instrumentation. This
    approach has worked superbly to provide the U.S. oceanographic
    community with the unquestioned leading sea-going capabilities for
    academic research. The issue at hand is whether the current
    management and funding paradigms will be suitable for emerging
    trends in oceanographic observational platforms and capabilities.
    The Facilities Programs under review have a traditional focus
    primarily on ships and the National Deep Submergence Facility. The
    programs under review make access to major ship based facilities
    nearly transparent for scientists with NSF funded research programs.

Looking at the developmental efforts within the science programs at NSF
and other funding agencies, it is apparent that within just a few years
the oceanographic community will also be using a range of additional
platforms and sensors on a community-wide basis. These platforms may
include drifters, gliders, cabled observatories, remotely operated
vehicles, and shallow depth submersibles, as just a few examples. These
systems do not currently fit comfortably for support within either the
science programs or operations programs. The challenge for NSF is to
position its programs (both acquisition of instrumentation and
operational support) to support these whole new classes of
community-based platforms, sensors and systems. Mechanisms are generally
not in place to enable coupled support between science proposals and new
facilities, as now exists to couple support between science proposals
and ships. Consideration should be given to developing a parallel
program to enable such transparent coupling, as has recently been done
with the Ocean Bottom Seismometer pool. Should these be supported
through a diversification of UNOLS responsibilities? Should NSF define
new program areas to support these emerging capabilities? Can the
current resources support these new capabilities in addition to the
existing set of platforms and facilities? We recommend that the Ocean
Sciences Division define this set of emerging community observational
capabilities and develop a strategy for their procurement, management
and support.

# 

NSF Response:

In looking to the future, the COV highlighted the changing nature of
facilities support in the ocean sciences, from the relatively simple
model of science supported by ships and submersibles, to a much broader
array of at-sea assets, including ships, submersibles, ROVs, AUVs,
cabled and buoyed observatories, drifters, gliders and potentially many
other systems. The COV believes that the Division will be challenged to
position its programs (in both acquisition and operation) to support
these new classes of platforms, sensors and systems. We agree with the
COV that it is necessary that "the Division of Ocean Sciences define
this set of emerging community observational capabilities and develop a
strategy for their procurement, management and support." Indeed, we have
already embarked on this journey, having developed a second generation
ROV (Jason II), AUV (ABE, the Autonomous Benthic Explorer), and are in
the process of working with the community in defining seafloor
observatories, their construction and management, and the U.S. Ocean
Observing System, through the National Oceanographic Partnership
Program. We also intend to seek the advice of the Ocean Studies Board on
the National Research Council on future needs for human-occupied
vehicles in relation to other technologies for the exploration of the
deep sea and seafloor.

**2004 NSF Update:**

The Ocean Studies Board study on deep submergence was commissioned and
the report completed. We indicated to the Board our intention of
adopting their recommendations and have made strides to implement them.
With regard to the Ocean Observatory Initiative (OOI), NSF has funded
the ORION project office to examine many of the issues noted in the COV
recommendation. Regarding facility acquisition, many developments since
the COV meeting have occurred, including: NSB approval of MREFCs for the
ARRV, International Ocean Drilling Program (IODP) and OOI; an MOA signed
with Naval Sea Systems Command (NAVSEA) for three Regional Class ships;
and separate cooperative agreements awarded to acquire a 3-D seismic
vessel, build a replacement for DSV ALVIN and build a full ocean depth
Hybrid Remotely Operated Vehicle (HROV).