**An Enhanced and Integrated**

**Strategic Planning and**

**Program Assessment Strategy**

for the

**National Sea Grant College Program**

![](media/image1.png){width="0.8861111111111111in"
height="0.8006944444444445in"}

Draft

Report of the

Sea Grant Response Integration Team

In response to the

National Research Council

(Ocean Studies Board)

Committee on the Evaluation of the Sea Grant Program Review Process

# Presented to

Dr. Leon M. Cammen

Director,

National Sea Grant College Program,

National Oceanic and Atmospheric Administration (NOAA)

Response Integration Team Members

Mr. Paul Anderson, President, Sea Grant Association

Dr. Nikola Garber, National Sea Grant Office

Dr. Jonathan Kramer, Past President, Sea Grant Association

Dr. James D. Murray, National Sea Grant Office

September 14, 2007

**Acknowledgements**

In compiling its report, the Response Integration Team (RIT) benefited
from the intellectual talent of a number of people with significant Sea
Grant experience. Major contributions were provided from representatives
of the Sea Grant Association, National Sea Grant Office and National Sea
Grant Review Panel. From the Panel, Ross Heath and Jerry Schubel
participated in an initial RIT committee which developed the RIT
framework and organized the 24 National Research Council recommendations
into three bundles of recommendations that could logically be addressed
by Sub-Teams. The members of three subsequently appointed Sub-Teams
contributed talent and time through a number of meetings and conference
calls over the past six months and provided the RIT with the ideas and
concepts that form the foundation of this report.

The members of the three Sub-Teams deserve particular credit and praise
for their contributions. The Sub-Team members include:

**Strategic Planning Process Sub-Team** - Barry Costa-Pierce (RISG),
Co-chair; Mike Voiland (NCSG), Co-chair; Megan Agy (NSGO); Jamie Krauk
(NSGO);

**Performance Assessment Sub-Team** - Russ Moll (CASG), Chair; Chuck
Wilson (LASG); Sami Grimes (NSGO); Terry Smith (NSGO)

**National Sea Grant Operations Sub-Team** - Jon Eigen (NSGO), Chair;
Dorn Carlson (NSGO); Don Scavia (MISG); Nancy Targett (DESG).

Drafts of the Sub-Team reports were distributed widely to the Sea Grant
network and the RIT received constructive comments on these drafts from
numerous individuals. These comments were most helpful for
groundtruthing initial ideas and concepts. The RIT expresses thanks to
each individual who took the time to respond to early drafts of the
Sub-Teams.

A committee report requires a good writer/editor to pull together the
component parts and to make the message coherent.  The RIT is grateful
to Amy Painter (NSGO) for lending her writing and editing talents and
for making this complex subject more understandable. 

**Table of Contents** Page

Acknowledgments 2

Table of Contents 3

Executive Summary 4

I.  Background 8

II. RIT Process 9

    a.  Integrating Planning and Program Assessment: The Vision 9

    b.  The RIT Structure 10

III. The Conceptual & Operational Framework 13

     a.  General Concepts

     b.  Strategic Planning *During* the Transition 15

     c.  Strategic Planning *During* the Transition 15

     d.  Metrics 17

     e.  NSGO Program Officer Roles and Interactions 17

     f.  Assessment Body and Processes 18

     g.  Additional Tools for Program Improvement and Recognition 20

### Comparison of Previous Performance Assessment Process and 22

> New Planning and Assessment Process

### Implications for the NSGO 24

###   {#section .unnumbered}

### Summary 26

###   {#section-1 .unnumbered}

*Appendices*

Appendix I - Planning and Program Assessment-Related Principles 27

that Guided the RIT

Appendix II -- Considerations for Developing the Content of Strategic
and 28 Implementation Plans (2013)

Appendix III - Duties of a Federal Program Officer 31

Appendix IV - Strategic Plan Alignment Memo Guidance 32

Appendix V - Strategic Plan Approval by the NSGO (2013) 33

Appendix VI - Topical Assistance Team (TAT) Guidance 35

Appendix VII - Program Review Panel 37

Appendix VIII - Mid-Course Review (2011) 40

Appendix IX -- RIT Response to Individual NRC Report Recommendations 42

## Executive Summary

The National Research Council (NRC) report, *Evaluation of the Sea Grant
Review Process* (2006), made recommendations for changes and
improvements to the process by which Sea Grant programs are assessed. In
order to address these recommendations, a Response Integration Team
(RIT), was formed to provide guidance to the National Sea Grant College
Program (NSGCP) Director, who is charged with responding to the NRC
report.

The RIT addressed all of the recommendations made by the NRC. However,
it was not the RIT's intent to recommend the adoption of each
recommendation independently and literally, but rather to develop an
integrated system for planning and assessment that accurately addresses
the intent of the entire suite of recommendations found therein.

This report presents the RIT's analyses, and offers guidance that will
be useful for both the National Sea Grant Office (NSGO) and the
university-based programs as the Sea Grant enterprise implements
measures to improve its strategic planning and assessment processes. The
new planning and assessment system described in this report builds upon
the former program assessment process and introduces several new
concepts that are designed to better integrate Sea Grant planning and
management to produce significant outcomes. In producing this report,
the RIT has tried to be inclusive and has utilized intellectual talent
from all parts of the network to conceptualize a holistic planning and
assessment process.

**Highlights**

The current Performance Assessment Team (PAT) process has been in place
for over 10 years. While several elements of program planning,
organization, assessment and management will continue under the new
integrated planning and assessment process, there are also several key
differences. The most significant difference between these two processes
is the elimination of the four-year PAT site-visit as the fundamental
component of Sea Grant program assessment. Highlights from the new
system include the following:

-   **Emphasis:** Programs evaluated for ability to *integrate*
    research, extension and education programming to address strategic
    issues and produce significant impacts

-   **Assessment:** PAT site-visit replaced by an independent Program
    Review Panel (PRP) to review all 31 programs concurrently, every
    five years, based upon briefing materials provided by programs. The
    process includes:

    -   Director participation: University-based program directors
        invited to address the PRP during review process to answer
        questions and provide additional information

    -   Mid-course review: Every two-and-a-half years, through PRP
        sub-group, largely to be based on aggregated annual reports

-   **Annual reports:** More closely linked to strategic plans

-   **Strategic plans:** Programs to align strategic plans with national
    plan

-   **Program visit:** National Sea Grant College Program Director
    visits programs once every five years (with program officer and
    National Sea Grant Review Panel member)

> Omnibus and outreach proposal cycles will remain the same. In
> addition, the new system recognizes the fundamental importance of
> ensuring programs the freedom and flexibility to respond to emerging
> issues and opportunities, as well as to critical state needs. These
> activities may extend beyond federal priorities and be funded by
> alternate means. The combination of these elements builds strength and
> credibility, and they may be reflected in program planning and
> reporting. Topical Assistance Teams (TATs) will become mandatory, and
> depending upon the expertise needed for a TAT, will involve National
> Sea Grant Review Panel members, along with other experts. As in the
> past, TATs will be based upon a program's desire to address
> deficiencies, or to foster program development and innovation. All
> TATs will be focused on program improvement.

**Timeline**

The RIT recognizes that the new model will have a number of important
implications with regard to the timing of critical actions on the part
of university-based programs, the National Sea Grant Office and National
Sea Grant Review Panel. Below, is a summary of each step in the process
over the next planning and assessment cycle.

**2008**

*National Strategic Plan*

A new national strategic plan will be completed by 2008. The national
priorities will reflect strong input from key stakeholders as well as
consensus on critical national needs from high-level analyses at state
and federal levels (bottom-up and top-down inputs).

*Initial Strategic Plan Alignment Process (beginning 2008)*

Alignment of state and national strategic plans will be an essential
process in the new planning and assessment system. During the transition
period, programs will be required to align their new or existing
strategic plans with the new national plan. Each Sea Grant program will
have time to examine the new national plan and to fully consider how its
current programming and existing strategic plan reflects (or does not
reflect) new national plan focus areas. Programs will be expected to
consider the goals and objectives of the 2008 national plan and whether
their current plan is responsive to those needs. In some cases, programs
may need to initiate some level of stakeholder engagement in order to
identify which national priorities best match to local needs and
interests. Approval by the NSGO will ensure that the aligned plan and
the intentions of the program are understood as the basis for the
upcoming program assessments.

*Alignment Memo*

All programs will be required to submit a written memo indicating how
they intend to align their current strategic plan with the new national
plan. This memo (up to five pages) will be due by April 1, 2008 after
programs have solicited staff, and some form of stakeholder input. Close
interaction between university-based program directors and their
assigned NSGO program officers will facilitate development of a strong
alignment memo, linking the state and national plans.

**2009**

*Initial Rating Review (one-time event)*

As the Sea Grant network transitions to a new planning and assessment
system, there is need to re-gauge existing program ratings in order to
update out-of-date ratings (some programs were last rated in 2003) and
implement a more comparative program rating structure. In early 2009,
all programs will be subject to re-rating by an independent panel of
experts. All programs will be reviewed over the same four-year review
period, which is defined as 2005-2008. Programs last reviewed as late as
2005 and 2006 will only need to update and modify their most recent
briefing book by adding information for the last two or three-year
period since the last assessment.

**2011**

*Mid-Course Review (ongoing, every two-and-a-half years)*

Although the NRC report emphasized the need for annual assessments, the
RIT believes that wholesale rating reviews of the programs on an annual
basis would be burdensome to all parts of the network. A mid-course
review will allow a program's rating to be adjusted either up or down
for demonstrated cause. This review will be conducted every
two-and-a-half years by a sub-group, comprised of five or six members
from the original 15-member Program Review Panel (PRP). This sub-group
of the PRP (PRP-sub) will be charged with reviewing new information
provided since the last PRP review and advising the NSGCP Director on
changes that should be made to the original 2009 program ratings.

**2012**

*Strategic Planning Process*

The entire process is re-initiated in 2012 as a new national strategic
plan is developed. Individual program planning will proceed in concert.

**2013**

*Strategic Plan Alignment*

All university-based program plans must be fully aligned with the
national strategic plan by this date. **By the year 2013, all
university-based programs will be on the same five-year planning
cycle.**

*Program Review (2013, and once every five years thereafter)*

A periodic and comprehensive review is critical to continued program
improvement and to the assessment of program performance. The PRP will
focus on the important impacts (both planned and those attained in
response to new opportunities) achieved within the planning framework
established at the state and national levels. This review will emphasize
the linkage between program impacts and management actions that
demonstrate strategic, integrated programming to achieve significant
results. The PRP will focus on evaluating the quality of the outputs,
outcomes and impacts as well as the management actions leading to them.

**Between 2008 and 2013**

The following will take place:

*Program Visits*

The NSGCP Director will visit each university-based program once during
the assessment cycle. This visit will include the program officer and a
member of the NSGRP and will be a brief visit to the host institution
for the primary purpose of meeting with the program management team,
advisory committees and university administration to review and discuss
the broad issues related to institutional setting.

*Topical Assistance Team (TAT)*

The new system includes one TAT for each university-based program during
the assessment cycle. The purpose of the TAT is largely for program
improvement and will be based upon needs identified by previous reviews
and/or the need for the program to identify and respond to new
opportunities.

**Annual/Biennial (ongoing)**

Annual reporting and omnibus cycles will remain the same.

**Conclusion**

The RIT believes that adoption of the processes outlined in this report
will encourage programs to be accountable to their strategic plans,
thereby meeting the needs of their stakeholders and demonstrating the
value of the federal-state partnership. Implementing these procedures
will position the program as a more nationally-focused, outcome-based
enterprise built "bottom-up" from its network of university-based,
locally relevant programs and strengthened "top-down" by its federal
mission.

**I. Background**

**National Research Council Report**

In 1993, the Under Secretary of Commerce for Oceans and Atmosphere
(National Oceanic and Atmospheric Administration) requested that the
National Academies' National Research Council (NRC) review and evaluate
the National Sea Grant College Program as part of an effort to prepare
for the National Sea Grant College Program Reauthorization Act of 1998
(P.L. 105-160). By the following year, the NRC completed its review and
produced a report entitled, *A Review of the NOAA National Sea Grant
College Program* (1994). In the report, the NRC recommended several
actions, including strengthening the strategic planning process at the
national level, clarifying the roles and responsibilities of the
National Sea Grant Office (NSGO) and the National Sea Grant Review Panel
(Panel), and carrying out systematic, periodic reviews of the individual
programs.

**Current NRC Study**

In response to the 1994 report, the Director of the National Sea Grant
College Program requested that the National Sea Grant Review Panel
establish a process for evaluating each individual program once over a
four-year review cycle. The Panel responded by establishing a program
assessment process in which Sea Grant program reviews are carried out
through a series of site visits involving recognized experts in marine
science and policy who focus on a uniform set of performance criteria,
using a standardized set of benchmarks and indicators. This assessment
process has evolved over time, in response both to experience gained
during its execution and to the evolving expectations of Congress. The
National Sea Grant College Program Act Amendments of 2002 (P.L.
107--299) directed NOAA to contract with the NRC to carry out a new
review of the assessment process and make appropriate recommendations to
improve its overall effectiveness.

**NRC Statement of Task**

The NRC's Committee on the Evaluation of the Sea Grant Program Review
Process was charged with assessing new procedures adopted by the NSGCP
since the publication of the 1994 NRC report. During this study, the NRC
Committee assessed the impact of the new procedures and evaluation
process on Sea Grant as a whole.

Specifically, the Committee was asked to examine:

1\. The effectiveness of major changes instituted in response to the
recommendations of the 1994 NRC report with regard to individual program
performance and quality.

2\. The effectiveness of program review procedures with regard to
accuracy, accountability, and enhancement of individual program
performance, including:

-   Review the effectiveness of the evaluation and rating system in
    determining relative performance of programs with regard to
    management and quality of research, education, extension and
    training activities;

-   Evaluate whether there have been improvements in programs as a
    result of the evaluation process;

-   Evaluate the 2003 review procedures for their ability to
    meaningfully segregate individual programs into five categories
    based on competitive scores; and,

-   Compare the effectiveness of the previous and 2003 review procedures
    with regard to the dual objectives of maximizing the quality of each
    program and of rating programs relative to each other for the
    purpose of determining performance-based funding.

3\. The usefulness and fairness of metrics developed to evaluate
programs with different operational constraints, resources and local
priorities, including:

-   Evaluate metrics for relevance and clarity;

-   Determine whether metrics provide a quantitative measure of quality
    of performance; and,

-   Assess whether metrics improve consistency and objectivity of
    reviews from different teams evaluating a diverse portfolio of
    individual Sea Grant programs.

The Committee was also asked to make recommendations for improving the
overall effectiveness of the evaluation process to ensure fairness,
consistency and enhancement of performance.

## Summary of NRC Recommendations

The NRC made a total of 24 recommendations in the following categories:
strategic planning; evaluation; periodic assessment and performance
criteria; program assessment teams and site visits; and, improving
program cohesion, coordination and oversight. This report addresses
those recommendations by presenting a new, integrated model for
strategic planning and assessment.

**II. Response Integration Team Process**

**Integrating Planning and Program Assessment: The Vision**

The NRC report recognized the NSGCP as a strong and effective national
partnership program. Through this unique partnership, the federal
government engages a network of 31 university-based programs to achieve
national goals. The outcomes and impacts resulting from Sea Grant
activities provide enormous ecological and economic benefits to the
public. The NRC report provides guidance and recommendations for how Sea
Grant can enhance its ability to make these important societal
contributions over time.

In order to develop a comprehensive and practical response to the NRC
recommendations, the NSGCP Director established a Response Integration
Team (RIT) comprised of members of the National Sea Grant Office (NSGO),
and the Sea Grant Association (SGA). The four-member RIT, with two
representatives from the SGA, two from the NSGO and with input from the
Panel, sought to develop a transparent process involving balanced input
from the Sea Grant network. The primary objective was to provide advice
to the Director about how best to respond to the NRC recommendations.
The NSGCP Director will make the final decision about a revised Sea
Grant planning and assessment system. This report presents the RIT's
analyses, and offers guidance that will be useful for both the NSGO and
the university-based programs as the Sea Grant enterprise implements
measures to improve its strategic planning and assessment processes.

The premise underlying this report is that by integrating Sea Grant
planning, management and assessment activities, the organization will
enhance its ability to produce critical coastal and Great Lakes impacts,
thus providing increased benefits to the public. This outcome-focused
system contrasts with the previous assessment system by placing greater
emphasis on the integration of planning and management to produce
significant impacts/outcomes. The proposed system encourages programs to
be accountable to their strategic plans, thereby meeting the needs of
their stakeholders, and demonstrating the value of the federal-state
partnership. The integrated planning and assessment system proposed in
this report will enhance Sea Grant's cost-effectiveness while
positioning the program as a more nationally-focused, outcome-based
enterprise built "bottom-up" from its network of university-based,
locally relevant programs and strengthened "top-down" by its federal
mission.

The RIT addressed all of the recommendations outlined in the NRC report
in the spirit in which they were offered. It was not the RIT's intent to
recommend the adoption of each NRC recommendation independently and
literally, but rather, to develop an integrated system for planning and
assessment based on the entire suite of recommendations.

**The RIT Structure**

As a mechanism for coordinating the National Sea Grant College Program's
response to the NRC Report, the RIT organized three Sub-Teams to address
the issues identified in the report. These included: a Strategic
Planning Process, a Performance Assessment Process, and a National Sea
Grant Office Operations Sub-Team. The membership of each Sub-Team
included two members of the NSGO, and two delegates of the SGA. Each
group elected a chairperson. The RIT also designated a liaison to each
Sub-Team to serve as the primary contact between the Sub-Team and the
RIT, and to ensure integration with other Sub-Teams. The RIT has
synthesized and incorporated many elements of the Sub-Team reports in
this document. The full reports for each Sub-Team are available on web
at:
[http://www.seagrant.noaa.gov/input/rit/](http://www.seagrant.noaa.gov/input/rit/ )

## Strategic Planning Process Sub-Team

The NRC identified strategic planning as one of the central elements for
consideration in the re-design of the program assessment process.
Strategic planning is a cornerstone of the NRC report and central to the
RIT's emphasis on planning for outcomes. Effective plans commit programs
to explicit outcome-oriented goals and objectives, and provide a
tangible link between state and national priorities.

The Strategic Planning Process Sub-Team provided guidance in two stages.
During the first stage, the team recommended a process, led by the NSGO,
to develop a National Sea Grant College Program Strategic Plan. During
the second stage, the team provided guidance to help university-based
programs strengthen their planning processes and align their plans to
the priorities articulated in the NSGCP Strategic Plan (national
strategic plan).

**Performance Assessment Process Sub-Team**

The NRC made a series of recommendations prescribing a new performance
review process, revised metrics and benchmarks, and a new rating and
assessment system. The Performance Assessment Process Sub-Team was asked
to provide analysis and guidance in these three critical areas.

Based on the principles outlined by the NRC, the Sub-Team developed a
set of recommendations to restructure the current program assessment
system. The recommendations include the creation of a program
improvement process and program assessment process. These two processes
are linked via the formation and activities of an independent Program
Review Panel that reviews all Sea Grant programs and provides initial
and ongoing assessment of each program.

**National Sea Grant Office Operations Sub-Team\
**The NRC identified the operations of the National Sea Grant Office as
one of the elements for consideration in the re-design of the program
assessment process. Effective two-way communication and guidance between
the NSGO and the Sea Grant programs is essential for success in
addressing state and national issues. All parties involved need to
understand their respective roles and responsibilities while committing
to open, transparent communication that enhances the effectiveness of
the partnership.

The Sub-Team analyzed the NSGO's workforce needs for strategic planning
and assessment, in part, based on the draft reports of the Strategic
Planning Process and Performance Assessment Sub-Teams. In its analysis,
the Sub-Team concentrated on the critical role that Program Officers
should play in linking individual state programs with the national
program through a coupled strategic planning/assessment model.

# 

**RIT Timetable**

**[2006]{.underline}**

**September:** RIT formed and initial meeting held

**November:** Sub-Team charges developed

**[2007]{.underline}**

**January:** Sub-Teams formed; initial conference calls

**February:** Initial RIT meetings with Sub-Teams; chairs elected,
Sub-Team schedules developed

**March:** Strategic Planning Process Sub-Team, Stage 1 Interim Report
to RIT and distributed to network for comment

**June:** Strategic Planning Process Sub-Team, Stage 2 Interim Report to
RIT

**June:** Draft Strategic Planning Process and Program Assessment
Sub-Team reports distributed to Sea Grant network for comment to RIT

**July:** RIT completes draft report and seeks comments from Sub-Teams

**August:** NSGO Operations Sub-Team Report to RIT

**August 28-30:** RIT Retreat: Sub-Teams and Panel NRC Response
Committee presented reports, gave feedback to the RIT on draft
integration document, RIT created draft report

**September:** RIT produces draft report and provides to the SG network
for comment

**October:** RIT presents draft report to the network at the SG
gathering

**October 15:** Comments provided to RIT on the draft report

**Late October:** RIT reviews comments, creates a final report

**November 1:** RIT presents final report to National Sea Grant College
Program Director

**RIT Principles**

In addressing the NRC's recommendations, and subsequently designing a
revised planning and assessment system, the RIT approached its charge
with one overarching principle: to do no harm to the Sea Grant program.
The full set of RIT principles is available in **Appendix 1** (page 27).
With this in mind, the RIT followed several specific principles that
were first described by the Strategic Planning and Performance
Assessment Sub-Teams (the full Sub-Team reports are available on the
web:
[http://www.seagrant.noaa.gov/input/rit/](http://www.seagrant.noaa.gov/input/rit/ ))

The approach and recommendations contained in this report are presented
as a set of actions that, taken as a whole, respond to the NRC
recommendations and provide a new, integrated approach for Sea Grant
planning and assessment. The RIT hopes that these concepts and ideas
stimulate critical thinking and innovation by the Sea Grant network as
it attempts to improve an already solid foundation of program planning
and assessment.

**National Sea Grant Review Panel Report and Advice**\
The NSGO Deputy Director served as liaison between the National Sea
Grant Review Panel (Panel) and the RIT, facilitating information flow as
both groups developed their responses to the NRC recommendations. All
RIT and RIT Sub-Team draft reports were provided to the Panel for
comment. The Panel was also provided with periodic updates on RIT
activities during monthly executive committee teleconferences.

The RIT utilized two draft Panel reports pertaining to program
assessment: *Review and Recommendations: Sea Grant Program Evaluation
Process Report of the Sea Grant Review Panel's Program Evaluation
Committee (November 2005)* and *Report and Recommendations of the NRC
Report Implementation Committee (February 2007).* The former report
provided a number of recommendations designed to improve the program
assessment process, while the latter addressed each of the 24 NRC
recommendations. In addition, Mr. Frank Kudrna, on behalf of the Panel's
NRC Report Implementation Committee, represented the Panel and shared
its views on the RIT's preliminary draft report during an August 28,
2007 RIT retreat.

While the conclusions and recommendations in this report are the
responsibility of the RIT, it should be noted that the Panel's response
efforts informed the RIT process. The RIT addressed each of the
evaluation issues of concern to the Panel with the exception of
developing procedures for re-competition, recertification,
decertification and re-designation of Sea Grant colleges and
institutional programs (as the Panel recommended in its November 2005
report). The RIT determined its first priority was to focus on
developing a comprehensive NRC response before the larger issue of
re-competition of programs could be addressed.

### III. The Conceptual & Operational Framework: Integrated Planning and Assessment  {#iii.-the-conceptual-operational-framework-integrated-planning-and-assessment .unnumbered}

### General Concepts {#general-concepts .unnumbered}

This model is derived from procedures employed over the past two program
assessment cycles. The proposed modifications and new evaluative
procedures reflect the recommendations made by the NRC and are based on
the need to more substantively link planning to outcomes at all levels
of the Sea Grant program.

**Timeline**

The RIT recognizes that the new model will have a number of important
implications with regard to the timing of critical actions on the part
of university-based programs, the NSGO and Panel. A chronology outlining
each step in the process over the next planning and assessment cycle
(2007-2013), is presented in **Table 1,** below. Details about each of
these components are discussed, below.

[]{.mark}

[]{.mark}

**Table 1. Timeline** **(Calendar Year 2007-2013)**

+-------+---------+---------+------+-------+-------+-------+---------+
|       | *       | *       | **20 | **2   | **2   | **2   | *       |
|       | *2007** | *2008** | 09** | 010** | 011** | 012** | *2013** |
+-------+---------+---------+------+-------+-------+-------+---------+
| **Uni |         | **State | # An | #     | #     | *     | **State |
| versi |         | St      | nual |       |       | **Bri | Stra    |
| ty-ba |         | rategic |  Rep | #     | #     | efing | tegic** |
| sed** |         | Plan    | ort  |       |       | Books |         |
|       |         | Al      |      | #     | #     | Pre   | **Plans |
| **    |         | ignment | *(3  |       |       | pared | Deve    |
| Progr |         | B       | ^rd^ | #     | #     | 20    | loped** |
| ams** |         | egins** | Q    |       |       | 08-20 |         |
|       |         |         | uart | #     | #     | 12*** | *Annual |
|       |         | ***Al   | er)* |       |       |       | Report* |
|       |         | ignment |      | #     | #     | *(    |         |
|       |         | Memo\   | #    |       |       | 4^th^ | *(3^rd^ |
|       |         | *       |      | #     | #     | Quar  | Qu      |
|       |         | *(2^nd^ |      |  Annu |  Annu | ter)* | arter)* |
|       |         | Qu      |      | al Re | al Re |       |         |
|       |         | arter)* |      | port  | port  |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ***B    |      | *     | *(    |       |         |
|       |         | riefing |      | (3^rd | 3^rd^ |       |         |
|       |         | Books   |      | ^Quar | Quar  |       |         |
|       |         | P       |      | ter)* | ter)* |       |         |
|       |         | repared |      |       |       |       |         |
|       |         | 2004-   |      |       |       |       |         |
|       |         | 2008*** |      |       |       |       |         |
|       |         | (4^th^  |      |       |       |       |         |
|       |         | Q       |      |       |       |       |         |
|       |         | uarter) |      |       |       |       |         |
+-------+---------+---------+------+-------+-------+-------+---------+
| ### P |         |         | **   |       | #     |       | **      |
| RP (P |         |         | *Ini |       | ### M |       | *5-year |
| rogra |         |         | tial |       | id-Co |       | Perf    |
| m Rev |         |         | Ra   |       | urse  |       | ormance |
| iew P |         |         | ting |       | Revie |       | Revi    |
| anel) |         |         | Rev  |       | w\*\* |       | ew\**** |
|  {#pr |         |         | iew\ |       |       |       |         |
| p-pro |         |         | **** |       | *(    |       | *(1^st^ |
| gram- |         |         |      |       | 2^nd^ |       | Qu      |
| revie |         |         | *(1  |       | Quar  |       | arter)* |
| w-pan |         |         | ^st^ |       | ter)* |       |         |
| el .u |         |         | Q    |       |       |       |         |
| nnumb |         |         | uart |       |       |       |         |
| ered} |         |         | er)* |       |       |       |         |
+-------+---------+---------+------+-------+-------+-------+---------+
| **N   | **N     | **N     | ##   | ##    | ##    | ## *  | **State |
| SGO** | ational | ational |      |       |       | Natio | Stra    |
|       | Stra    | Stra    | ##   | ##    | ##    | nal S | tegic** |
|       | tegic** | tegic** |      |       |       | trate |         |
|       | **Plan  |         | ##   | ##    | ##    | gic P | *       |
|       | Develo  | **Plan  |      |       |       | lan C | *Plan** |
|       | pment** | Comp    | ##   | ##    | ##    | omple |         |
|       |         | leted** |      |       |       | ted*  | **App   |
|       |         |         |      |       |       |       | roval** |
|       |         | **State |      |       |       | ##    |         |
|       |         | Stra    |      |       |       |       |         |
|       |         | tegic** |      |       |       | ##    |         |
|       |         |         |      |       |       |       |         |
|       |         | **Plan  |      |       |       |       |         |
|       |         | App     |      |       |       |       |         |
|       |         | roval** |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ##      |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ## TAT  |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ## Lead |      |       |       |       |         |
|       |         | ership  |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ##      |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | #       |      |       |       |       |         |
|       |         | # PO &  |      |       |       |       |         |
|       |         | NSGCP D |      |       |       |       |         |
|       |         | irector |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ***Vi   |      |       |       |       |         |
|       |         | sits*** |      |       |       |       |         |
+-------+---------+---------+------+-------+-------+-------+---------+
| ### N |         | *\      |      | ***   |       | ***   |         |
| SGRP  |         | **      |      | State |       | State |         |
| {#nsg |         | Program |      | of    |       | of    |         |
| rp .u |         | Visit   |      | Sea   |       | Sea\  |         |
| nnumb |         | w/***   |      | Grant |       | Grant |         |
| ered} |         | *       |      | Repo  |       | Repo  |         |
|       |         | **NSGCP |      | rt*** |       | rt*** |         |
|       |         | Dir***  |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | **      |      |       |       |       |         |
|       |         | *TAT*** |      |       |       |       |         |
|       |         |         |      |       |       |       |         |
|       |         | ***Pa   |      |       |       |       |         |
|       |         | rticipa |      |       |       |       |         |
|       |         | tion*** |      |       |       |       |         |
+-------+---------+---------+------+-------+-------+-------+---------+

> \*PRP-Full; The 15 member Program Review Panel (PRP) convened to rate
> all programs
>
> \*\*PRP-Sub: A smaller (5-7 member) set of PRP members convened to
> consider re-rating all programs and to address petitions from programs

**National Strategic Plan (2008)**

A network-wide strategic planning process (currently underway) will
identify a set of national priorities---areas in which Sea Grant is
uniquely qualified and positioned to make large-scale contributions to
issues of national need. As outlined by the Strategic Planning Process
Sub-Team in its Phase I report (for the full report, visit:
[http://www.seagrant.noaa.gov/input/rit/](http://www.seagrant.noaa.gov/input/rit/ )),
the priorities will reflect strong input from key stakeholders as well
as consensus on critical national needs from high-level analyses at
state and federal levels (bottom-up and top-down inputs). The
university-based programs should find most of these priorities relevant
and timely in their given states and regions. Strong participation in
the planning process (by the programs) will help ensure willingness to
work within the broad context set by these priorities.

**Strategic Planning *During* the Transition**

**Initial Strategic Plan Alignment Process (beginning 2008)**

Strategic planning at the state and national levels is central to the
new program planning and assessment process. During the transition
period, programs will be required to align their new or existing
strategic plans with the new national plan. Each Sea Grant program will
have time to examine the new national plan and to fully consider how its
current programming and existing strategic plan reflects (or does not
reflect) new national plan focus areas. Programs will be expected to
consider the goals and objectives of the 2008 national plan and whether
their current plan is responsive to those needs. In some cases, programs
may need to initiate some level of stakeholder engagement in order to
identify which national priorities best match to state needs and
interests. **Programs due to submit a new strategic plan** will simply
need to consider the content of the new national plan as they go through
their new planning process. **Programs whose plans still have more than
a year remaining** will be asked to modify their plans to address
national priorities in an appropriate manner.

**Strategic Plan Alignment Memorandum and NSGO Acceptance (2008)**

All programs will be required to submit a written memo indicating how
they intend to align their current strategic plan with the new national
plan. This memo (up to five pages) will be due in 2008 after programs
have solicited some form of staff and stakeholder input. The memo will
be directed to the NSGCP Director with copies to the program officer.
Close interaction between each university-based program director and the
assigned NSGO program officer will facilitate development of a strong
alignment memo, linking the state and national plans. Approval by the
NSGO will ensure that the aligned plan and the intentions of the program
are understood as the basis for the upcoming program assessments.
**Appendix IV (**page 32), provides guidance for developing this memo,
and explains the acceptance process.

**Strategic Planning *After* the Transition**

**By the year 2013, all university-based programs will be on the same
five-year planning cycle.**

**Alignment of Priority Areas and Assessment**

Identification of national priority areas will remain the basic
foundation for each Sea Grant program's plan. By 2013, each
university-based program's strategic plan must be aligned with the
priorities articulated in the national strategic plan.

The intersection of state and national strategic planning represents a
critical juncture between planning and assessment. At this point, the
state-to-national linkages are established, and the long-term
(five-year) plan, against which the program will be evaluated, is set
into motion. This alignment is a critical step in establishing the
context for assessment---providing the foundation for how evaluators
will determine if outcomes from a given state are making contributions
to national priorities, and how the program is managing its resources.

Each program should be able to address a "critical subset" of the
national strategic priorities. Programs must determine which priority
areas are most relevant given state needs, drivers and resources, and
must explain the context for addressing the particular national
priorities selected.

**State Priorities**

Because many programs utilize and rely upon a mix of funds that extend
beyond Omnibus NOAA Sea Grant funding and match, program plans can (and
perhaps should) include additional priorities that reflect the diverse
nature of support as well as critical needs within their respective
states. A state strategic plan should accurately represent the diverse
and unique challenges each program will address and emphasize. The plan
should also articulate how resources, engagement, partnerships and
entrepreneurial management will be utilized to develop impacts. In this
manner, the plan is a true representation of the essential value of the
federal-state partnership that underlies Sea Grant nationwide.

**Emerging Issues and Opportunities**

Although planning is critical to producing a set of anticipated outcomes
and impacts upon which programs may be evaluated, there should also be
provisions within each state strategic plan allowing programs the
flexibility to respond to emerging issues and opportunities. Flexibility
and responsiveness are among Sea Grant's greatest strengths. These core
functions should be recognized as critical elements in strategic plans,
and in turn, considered during the assessment process.

**Planning Process**

Programs will be expected to demonstrate a strong planning process.
Guidance developed by the Strategic Planning Process Sub-Team outlines a
robust planning process that includes endorsement by appropriate
stakeholders and university administration, and is designed to produce
plans that are sufficiently detailed and appropriately challenging. This
process is based on input from a variety of well-established strategic
planning protocols. **Appendix II** (page 28) provides details of the
expectations for a successful strategic plan.

**Strategic Plan Approval by the NSGO (2013)**

The programs, NSGO and the assessment body must understand why goals
have been set, that they are sufficiently ambitious within the context
of resources and peers, and that they reflect the ongoing balance
between state and national drivers that shape each program. Programs
must be able to justify the critical choices they have made and the
context within each state. Close interaction between each program
director and the assigned NSGO program officer will facilitate
development of a strong plan and will be critical to the approval
process. **Appendix V** (page 33) elaborates on the approval process.

**Implementation Planning**

Effective implementation planning articulates how resources will be
utilized, and is therefore essential in this model. Programs may adopt a
number of different formats and approaches for implementation planning.
For instance, some programs may utilize integrated strategic and
implementation plans, while others may separate these into different
documents. Regardless of the model adopted, each program must articulate
how its portfolio of resources (research, outreach and education) will
be used to address strategic goals over specific timeframes.
University-based program implementation plans should contain detailed
descriptions of specific actions and timelines for meeting goals and
objectives within the context of what is most important for the program
in its state niche. The intersection between thematic and functional
aspects of the programs should also be evident. Implementation plans may
also articulate how other resources extending beyond Omnibus funding
(i.e., leveraged funding, partnerships, etc.) will be deployed.

**Metrics**

A considerable amount of effort has gone into defining metrics for the
NSGCP over the past decade. This has occurred at both the national and
university levels, where detailed implementation benchmarks and
appropriate metrics have been institutionalized. Further guidance will
be developed as part of the national strategic planning process to
provide a set of national metrics and benchmarks. It is possible that
some of these metrics will be derived directly from the previous PAT
process, while others may be intended to help assess progress on
localized goals and objectives, as articulated in state strategic plans.
Programs will need to describe how they are working to meet these
benchmarks with specific timelines, outputs and outcomes.

**NSGO Program Officer Roles and Interactions**

Program officers (POs) should be well-versed in how the Sea Grant
programs are implementing their strategic plans. This "front-line"
interaction between the NSGO and university-based programs must be
maintained and enhanced. A number of options exist to strengthen this
engagement, along with greater opportunities for joint learning and
exchange, in a shared process for program improvement and
accountability.

**Assessment Body and Processes**

**Program Review Panel**

A periodic and comprehensive review is critical to continued program
improvement and to the assessment of program performance. The first
performance review for the individual Sea Grant programs under this new
planning and assessment process is proposed as a five-year review.
Rather than basing this process largely on a site visit, as in the past,
the RIT proposes that all programs be reviewed by a Program Review Panel
(PRP). The PRP is the primary mechanism for comparison among programs
and will therefore provide justification to the NSGO for the assignment
of merit funds. A detailed description of the PRP and its operations is
found in **Appendix VII** (page 37).

The NSGCP Director will appoint 15 members to the PRP representing
varying expertise (relevant research, outreach and education
backgrounds) and a strong knowledge of Sea Grant.

The PRP will focus on the important impacts (both planned and those
attained in response to new opportunities) achieved within the planning
framework established at the state and national levels. This review will
emphasize the linkages between program impacts and management actions
that demonstrate strategic, integrated programming to achieve
significant results. The PRP will focus on evaluating the quality of the
outputs, outcomes and impacts, as well as the management actions leading
to them. In doing so, the PRP must consider, understand and evaluate how
well individual programs have met the goals of their strategic plans.
The PRP will examine impacts and significant contributions from the
state to national levels and their relevance to priorities articulated
in the state and national strategic plans. The PRP will also recognize
how emerging issues, opportunities and state-priorities are addressed.
In this manner, there is continuity between this process and the former
program assessment process.

**State Program Director Input to PRP**

Directors from each Sea Grant program will have the opportunity to meet
with the PRP in order to address issues and answer questions.

**Initial Rating Review (2009)**

As the Sea Grant network transitions to a new planning and assessment
system, there is need to re-gauge existing program ratings in order to
update out-of-date ratings (some programs were last rated in 2003) and
to implement a more comparative program rating structure. In early 2009,
all programs will be subject to re-rating by the PRP. All programs will
be reviewed for the same review period: 2005-2008. Programs last
reviewed in either 2005 and 2006 will only need to update and modify
their most recent briefing book by adding information for the last two
or three-year period. The PRP will assess all 31 programs based on
written materials provided in the briefing book, and the program's
response to recommendations from the most recent program assessment. POs
should be available to answer questions or provide clarification on
issues that arise. The PRP will provide solid evidence and rationale for
its findings and conclusions, and will clearly articulate what a program
should consider to improve upon a rating in a particular area.

**Mid-Course Review (2011)\
**The NRC report emphasized the need for annual assessments, in part to
recognize and reward programs for improvements made on a more regular
basis, and in part to de-emphasize the event nature of the previous
program assessment system. Although accountability and outcomes will be
tracked annually through annual reports, the RIT believes that wholesale
rating reviews of the programs on an annual basis would be burdensome to
all parts of the network. A mid-course review is proposed, through which
a program's rating could be adjusted either up or down for demonstrated
cause. This review will be conducted by a mid-course review sub-group,
comprised of five or six members from the original 15-member PRP. This
sub-group of the PRP (PRP-sub) will be charged with reviewing new
information provided since the last PRP review, and advising the NSGCP
Director on changes that should be made to the original 2009 program
ratings. See **Appendix VIII** (page 40) for further details.

###### Program Review (2013)

The PRP will utilize a set of documents generated over the lifespan of
the strategic planning process to assess each program. Of particular
importance are the briefing book and annual reports (described below)*.*
Additional materials will include: strategic and implementation plans,
any mid-course correction documented during the review period, and any
mid-course rating review documents.

> ***Briefing Books***
>
> The development of briefing books has become a catalyst for programs
> to examine program accomplishments and management, and to organize and
> present these in a logical, understandable manner. The preparation of
> a briefing book is central to the RIT's proposed system, and is an
> essential tool that will be used to examine the long-term outcomes and
> impacts of planning and management. The book should be derived in
> large part from annual reports. Therefore, programs should seek ways
> to organize annual reports so that the most relevant information could
> easily be incorporated into briefing books. The briefing book will
> have a standardized format with requirements outlined for both the
> program and the PRP's benefit. Ideally, programs should also have some
> freedom to demonstrate their individual character in these documents.
> Programs will likely prepare these briefing books during the final
> quarter prior to the review year. The briefing materials should
> demonstrate how the program has achieved strategic outcomes and
> impacts articulated in its most recent strategic plan, and how the
> integration of the program's assets has resulted in impacts at the
> state, regional and national levels.
>
> ***Annual Reports***
>
> At present, individual programs submit annual reports to the NSGO.
> These documents have evolved over time to compile impacts and broad
> contributions to NOAA priority areas. In the proposed system, annual
> reports take on a new role, linked much more closely to the strategic
> planning process at the state and national levels. Each program will
> examine progress toward its stated goals and benchmarks on an annual
> basis, and collate this information. The annual report will describe
> each state program's progress toward implementing and achieving its
> strategic plan goals and objectives. These reports should demonstrate
> how actively management is engaged, and show that outcomes and impacts
> are being tracked to both the state and national strategic plans. The
> annual report is an essential point of intersection between planning
> and assessment.

**The entire process will be re-initiated in 2012** as a new national
strategic plan is developed. See **Table I** (page 14). Individual
program planning will proceed in concert.

**By the year 2013, all university-based programs will be on the same
five-year planning cycle.**

**Additional Tools for Program Improvement and Recognition**

The NRC emphasized the need to continuously seek ways to improve
programs on all levels. The following mechanisms: a site visit by the
Director of the National Sea Grant College Program, and Topical
Assistance Team (TAT) visits, are resources available to the national
and university-based programs to aid in self-assessment and improvement.

###### Program Visit (NSGCP Director)

An active, vital link should exist between the state program, its NSGO
program officer and the leadership of the NSGCP. Enhanced information
exchange among these parties is essential to program improvement.
Similarly, engaging members of the National Sea Grant Review Panel in
continuous learning about programs stands to benefit the network as a
whole. Once every five years, the NSGCP Director, the assigned PO and a
Panel member will visit each Sea Grant program. Aside from the benefits
associated with increased knowledge and information exchange, these
visits will provide university-based program directors an opportunity to
showcase their programs to senior university administrators, faculty and
stakeholders. In addition, the NSGCP Director, by seeing each program
and understanding its strengths, weaknesses and subsequent rating, will
be able to appraise the success of the overall rating system.

This one or one-and-a-half day visit will involve Sea Grant program
staff, university officials and advisory bodies. The criteria set forth
in the designation of Sea Grant College Status, as described in the Code
of Federal Regulations (Guidelines for Sea Grant Colleges--CFR, Vol. 44,
No. 244; 12/18/7915 CFR part 918.3) could be used as a framework for
fact-finding during this visit. These criteria include: leadership,
relevance, programmed team approach, education and training, advisory
services, relationships, productivity and support. The NSGCP Director
will visit approximately six programs per year. Each program should
expect one visit during the course of a five-year review period. The
director of each program, in consultation with the NSGO PO, will set the
agenda.

**Self-Assessment and Realizing Emerging Opportunities**

**Topical Assistance Team (TAT)**

Effective programs "live" good strategic plans and commit to learning
from them in an iterative manner. Self-assessment therefore, is an
important tool to use throughout a planning interval. This process can
signal where corrections need to be made if benchmarks are not being
met, and help management determine how functional elements are working.
Similarly, self-assessment can indicate whether or not new opportunities
should be addressed.

While many programs have ideas and potential mechanisms to make
improvements, they may need assistance in addressing opportunities or
challenges---either thematic or functional. Programs also stand to
benefit from outside experts who bring independent thinking, new ideas
and a different kind of credibility to the process. The TAT approach has
been viewed widely as a successful mechanism to help in this way.

Programs could identify an area(s) that needs improvement or an
opportunity that could be realized with the help of a TAT. The program's
PO and a National Sea Grant Review Panel member (representing an
appropriate area of expertise) would participate on the teams. A TAT
visit would be required in a given planning cycle. The TAT report would
be of primary use to the program. However, the state program director
would have the option of sharing this information with the PRP.

***State of Sea Grant* Report**

The planning and assessment process described here will provide a wealth
of information pertinent to the national role and stature of Sea Grant,
and will reflect upon how the program is managed at the national level.
An in-depth analysis of this material should be done on a biennial
basis.

In lieu of the biennial report the NSGO currently publishes, the NSGRP
will create a *State of Sea Grant Report* that will be presented to key
audiences, stakeholders and decision-makers. The report will document
the overall program's progress toward achieving national strategic plan
goals and objectives. Much of the information should be aggregated from
state program annual reports, briefing books and selected materials from
the PRP process. The Panel should lead this analysis and the preparation
of the final report. Just as individual program briefing books, annual
reports and other synthesis materials provide the data and analysis that
should inform a new state planning process, the *State of Sea Grant
Report* should inform the new national planning process.

**IV. Comparison of Previous Performance Assessment Process and New
Planning and Assessment Process**

The current Performance Assessment Team (PAT) process has been in place
for over 10 years. Each of the university-based programs has been
reviewed twice under this process. While several elements of program
planning, organization, evaluation and management will continue under
the new integrated planning and assessment process, some components
differ. The chart below summarizes the differences and similarities in
the major elements of the former and proposed processes.

The overall consequence of this new, integrated approach to program
planning and assessment is to place greater emphasis on the end-to-end
linkages between planning and achieving strategic outcomes through
integrated programming. Where the former program assessment system was
based upon the discreet evaluation of management, implementation
processes, and producing significant results; the new system will assess
programs based on their *ability to integrate* research, extension and
education programming to effectively address strategically-identified
issues or problems. The new system will recognize program management's
ability to link program implementation to strategic planning and will
also allow programs the flexibility to address emerging issues.

**Differences**

The most significant difference between these two processes is the
elimination of the four-year PAT site-visit as the fundamental component
of Sea Grant program assessment. Under the new process, the PAT
site-visit is replaced by a peer-review approach that utilizes a Program
Review Panel (PRP) to review all 31 programs concurrently, every five
years, based upon briefing materials provided by the programs (described
above). The Sea Grant directors (or designees) will be invited to
address the PRP during this review process in order to answer questions
and to provide additional information for consideration by the PRP.

Another important difference is the increased reliance on annual
reporting under the new process. Although the format and information
required in the new annual reports are yet to be determined, there will
be changes to annual reporting that ensure linkage to strategic plans.
As mentioned above, the information included in annual reports will
aggregate over time, and will provide the primary basis for mid-course
rating reviews and formal program assessment every five years by the
PRP.

The university-based programs will need to align their strategic plans
with the national strategic plan. In addition, depending on the current
status of their strategic plans, programs will need to describe how
their plans will be aligned with the national plan at present and in the
future.

Whereas under the past PAT process, 25% of the programs are rated
annually (and some, by default, are moved into a different category in
the ranking system), the proposed process intends to review the ratings
of all programs at mid-cycle (every two-and-a-half years) through the
utilization of a Sub-team of the PRP (PRP-sub).

The PAT process rated all programs once every four or five years, while
the new process will provide more frequent, mid-course ratings through
the PRP-sub (based largely upon aggregated annual reports).

Another new element to this process is a visit to each university
program once during the five-year period by the NSGCP Director,
accompanied by the PO and a member of the Panel.

**Common Elements**

Although the linkage to the new national strategic plan provides a new
context for university-based program planning, the need for a strategic
plan and a scheme for implementing that plan remains the same for both
of these processes. Omnibus cycles and outreach proposal processes will
remain the same, as will each program's ability to include other,
discretionary, non-Sea Grant activities in its planning and reporting.
Topical Assistance Team (TAT) visits will become mandatory and will
involve panel members along with other experts. However, the TATs will
be based on program improvements, innovations or proposed developments.

**Table 2. A Summary of Differences and Similarities in the Major
Elements of the Former and Proposed Processes**

+---------------------+----------------------+------------------------+
| **OLD**             | **NEW**              | **Change**             |
+---------------------+----------------------+------------------------+
| **Previous Program  | **Integrated         |                        |
| Assessment          | Planning and         |                        |
| Process**           | Assessment Process** |                        |
+---------------------+----------------------+------------------------+
| Strategic and       | Strategic and        | Formalized alignment   |
| Implementation      | Implementation Plans | with national          |
| Plans Required      | Required             | strategic plan         |
|                     |                      | required               |
| Plans evaluated as  |                      |                        |
| part of program     |                      | Approval by NSGO       |
| assessment process  |                      | required               |
|                     |                      |                        |
|                     |                      | All Plans Synchronized |
|                     |                      | starting in 2013       |
|                     |                      |                        |
|                     |                      | Plans are not          |
|                     |                      | evaluated in           |
|                     |                      | assessment process     |
+---------------------+----------------------+------------------------+
| Omnibus             | Omnibus              |                        |
|                     |                      |                        |
| 2-year research     | 2-year research      |                        |
| proposals           | proposals            |                        |
|                     |                      |                        |
| 4-year outreach     | 4-year outreach      |                        |
| proposals           | proposals            |                        |
+---------------------+----------------------+------------------------+
| Other non-Sea Grant | Other non-Sea Grant  |                        |
| activities,         | activities, funding, |                        |
| funding, partners   | partners and         |                        |
| and regional        | regional approaches  |                        |
| approaches          | acknowledged and     |                        |
| acknowledged and    | expected             |                        |
| expected            |                      |                        |
+---------------------+----------------------+------------------------+
| Annual Reports      | Annual Reports with  | Greater links to       |
|                     | new criteria and     | planned                |
|                     | emphasis             | accomplishments (state |
|                     |                      | to national) to be     |
|                     |                      | detailed in new Annual |
|                     |                      | Reports                |
+---------------------+----------------------+------------------------+
| PAT site visit      | Program Review Panel | No PAT site visit      |
|                     | (PRP)                |                        |
|                     |                      | Program Review Panel   |
|                     | 5-year program       | (PRP) replaces PAT     |
|                     | assessment           |                        |
|                     |                      | 5-year program         |
|                     | Mid-course review    | assessment             |
|                     |                      |                        |
|                     |                      | Mid-course review      |
+---------------------+----------------------+------------------------+
| Briefing Books      | Briefing Books       |                        |
+---------------------+----------------------+------------------------+
| Topical Assistance  | Topical Assistance   | Topical Assistance     |
| Team visits (TAT)   | Team visits (TAT) -- | Team visits (TAT) --   |
| as needed           | required once every  | required once every 5  |
|                     | 5 years              | years                  |
+---------------------+----------------------+------------------------+
| Rolling rating      | Mid-course review    | Rolling rating scheme  |
| scheme over 4-5     |                      | over 4-5 year period   |
| year period         |                      | eliminated             |
|                     |                      |                        |
|                     |                      | Mid-course review      |
|                     |                      | instituted             |
+---------------------+----------------------+------------------------+
| *Occasional* NSGRP  | NSGCP Director       | NSGCP Director visits  |
| Director visits     | visits to programs   | to programs required   |
|                     | required once every  | once every five years  |
|                     | five years           |                        |
+---------------------+----------------------+------------------------+

###### 

**V. Implications for the NSGO**

The National Sea Grant Office (NSGO) is central to the successful
implementation of recommendations made by the RIT. Throughout its
report, the NRC Report called for an increased role in planning and
assessment by NSGO Program Officers (POs). As a result of budgetary
constraints and staff shortages, the NSGO has lost PO capabilities in
recent years.

A key question that must be addressed is whether or not the NSGO is
presently organized or could be organized to implement this proposed
planning and assessment system. Thus, to properly emphasize the vision
of this document, the NSGO must be organized to highlight the critical
role Program Officers (POs) play in linking individual university-based
programs with the National Sea Grant College Program and NOAA via
outcomes through planning. With proper staffing and organization, POs
will communicate the needs of the ocean, coastal and Great Lakes
communities to NOAA and identify opportunities for the Sea Grant network
as new programs within the Agency are identified and developed.

**Program Officer**

The integration of strategic planning, accountability and assessment
systems for the NSGCP relies upon POs to perform four main functions:
program oversight, grants management, program planning and program
assistance. Other functions of the NSGO critical to the successful
operation of the NSGCP include: leading and coordinating activities at
the national level (extension, education, communication, research and
fiscal); communicating on a national level with internal and external
stakeholders; managing national fellowship and strategic investment
programs; supporting program development at NOAA; and completing
performance and accountability requirements expected of all NOAA
Programs. To learn more about the duties of a federal PO, see **Appendix
III** (page 31).

**Current and Ideal Workforce**

The NSGO had 20 full-time equivalents (FTE) and three contractors as
recently as 2005. Budget cuts over last two years have resulted in a
corresponding loss in capacity. The NSGO has responded to this lost
capacity in several ways, by: (1) reducing the amount of time spent on
lower-priority activities while preserving, to the extent possible, the
capacity to perform its highest-priority activities (providing PO
support to university-based programs, and responding to NOAA and federal
program requirements); (2) partnering with other NOAA programs (e.g.,
Aquaculture, Invasive Species) to achieve goals common to both programs
at reduced cost; and, (3) using contractors, detail positions and more
junior federal employees for tasks previously performed by more senior
federal employees.

As of the end of FY 2007, the NSGO currently has 15 staff (11 FTEs,
three contractors/IPAs, and one FTE detailed by NMFS). In addition,
there are three other individuals who spend about 20% of their time on
Sea Grant activities, but whose salaries are supported elsewhere. With
the current budget, the NSGO can support a staff of 14 FTEs dedicated to
Sea Grant activities.

Based on the information provided by the NSGO Operations Sub-Team, an
ideal staff for the NSGO would be approximately 27 FTEs (or non-Federal
equivalents). For this level of staffing to occur, additional funding
opportunities need to be sought (i.e., increased Sea Grant
appropriations or funds provided by OAR, NOAA or DOC).

In the meantime, the staffing must continue to be arranged to meet the
current statutory and regulatory requirements. This includes a director,
a Designated Federal Official (for FACA responsibilities), certified
federal program officers sufficient to provide oversight and grants
management, a fiscal officer, and planning and assessment staff. The
next staffing priority should be to hire individuals to provide *all* of
the PO functions described. New hires should be POs with collateral
functional area duties (i.e., PO with Extension Director
responsibilities).

**Costs**

In the previous program assessment system, the NSGO funded PATs, parts
of non-mandatory TATs and occasional NSGRP Director visits. In the
proposed new system, the NSGO would fund mandatory TATs, the five-year
program assessment utilizing a Program Review Panel, the mid-course
review and the Director\'s visit. In comparing the approximate costs of
the two systems, the estimated cost difference between the two systems
was found to be minimal. Given the above information, the RIT believes
the proposed system can be accomplished with the current staff. Without
additional resources, the NSGCP Director will need to be willing to make
difficult choices, and must be prepared to reduce or eliminate other non
PO related activities.

**Summary**

The new planning and assessment system described in this report builds
upon the former program assessment process and introduces several new
concepts that are designed to better integrate Sea Grant planning and
management to produce significant outcomes. In producing this report,
the RIT has tried to be inclusive and has utilized intellectual talent
from all parts of the network to conceptualize a holistic planning and
assessment process that responds to the major recommendations of the NRC
report. Adoption of the processes outlined in this report will encourage
programs to be accountable to their strategic plans, thereby meeting the
needs of their stakeholders, and demonstrating the value of the
federal-state partnership. Implementing these procedures will position
the program as a more nationally-focused, outcome-based enterprise built
"bottom-up" from its network of university-based, locally relevant
programs and strengthened "top-down" by its federal mission.

**Appendix I**

Planning and Program Assessment-Related Principles

that Guided the RIT

**Planning-related principles**

-   Include planning for both thematic (national ocean, coastal, Great
    Lake priorities) and functional (research, education, outreach)
    areas;

-   Include input of stakeholders before the plan is finalized;

-   Capture the "strategic planning landscape" by conducting both
    top-down and bottom-up analysis by reviewing existing national and
    state plans and priorities before finalizing a national plan;

-   Make connections to other ocean/coastal/Great Lake agencies and
    stakeholders, and articulate timelines for completion of a national
    plan;

-   Explain how the national planning process was "strategic" by
    elaborating upon the specific strengths, weaknesses, opportunities
    and niches for Sea Grant;

-   Conduct gap and risk analyses in order to identify those priorities
    and niches; and,

-   Recognize that the continuum of planning stems from a robust
    national strategic plan to a solidly aligned state strategic plan to
    a supportive and clear implementation plan---an implementation plan
    with clearly articulated milestones and expected outcomes that can
    easily be tracked for annual reporting and assessment, along with a
    process for the regular evaluation and revision of the strategic
    plan.

**Program assessment-related principles**

-   At its core, the program assessment system should be designed to
    improve programs;

-   To the extent possible, the performance system should be collegial
    and not adversarial;

-   The program assessment system should provide accountability at the
    program, regional and national levels;

-   The program assessment system should provide increased
    responsibility and an enhanced role for program officers;

-   The program assessment system begins with integrated strategic and
    implementation planning and flows, in part, from benchmarks and
    milestones resulting from the planning process;

-   Program assessment should relate to planned accomplishments;

```{=html}
<!-- -->
```
-   The program assessment system should account for a program\'s
    regional and national contributions;

-   The program assessment system should be completely transparent; and,

-   The program assessment system should be as simple as possible.

# **Appendix II** 

# 

# 

# Considerations for Developing the Content of Strategic and Implementation Plans (2013)

Strategic and implementation planning are management tools. As such,
they should help an organization do a better job. Strategic plans
provide a clear focus of energy, ensure that members of the organization
are working toward the same goals, and outline clear processes to assess
and adjust the organization\'s direction in response to a changing
environment. In short, strategic planning is a disciplined effort to
produce fundamental decisions and actions that shape and guide what an
organization is, what it does, and why it does it, with a focus on the
future. (Adapted from *Bryson\'s Strategic Planning in Public and
Nonprofit Organizations)*

**A. Principles of Planning**

In conducting strategic planning, it is recommended that the
university-based programs fully utilize an inclusive framework that
tracks to the national strategic planning process and is tailored to
meet the specific landscape in each state. A well designed strategic
plan:

-   Includes both thematic and functional areas;

-   Includes extensive input of stakeholders before the plan is drafted;

-   Captures the "strategic landscape" by reviewing existing national
    and state plans and priorities;

-   Articulates the processes used to determine priorities;

-   Makes connections to other agencies and stakeholders, and
    articulates timelines for completion;

-   Explains how the process was "strategic" by elaborating upon the
    specific opportunities and niches;

-   Conducts gap and risk analyses in order to identify those priorities
    and niches;

-   Includes the recognition that the continuum of planning stems from a
    robust national strategic plan, to a solidly aligned state strategic
    plan;

-   Includes clearly articulated milestones and expected outcomes that
    can easily be tracked for annual reporting and evaluation; and,

-   Incorporates a process for the regular evaluation and revisions of
    state plans.

**B. The Value of Stakeholders in Strategic Planning**

The most effective programs utilize a sound strategic planning framework
that includes input and advice from a broad array of
stakeholders---including user, constituency and advisory groups---at
appropriate levels and phases throughout the planning process. The
strength of Sea Grant at the state and regional levels is in the
diversity of stakeholders who are served by the university-based
programs, and who are able to provide the programs with sound advice.
Effective and ongoing collaboration with all stakeholders in the plan's
development, approval, implementation, monitoring and evaluation is
required to ensure that the plan is based upon, and reflects priority
needs at the local, state and regional levels.

**C. Types of Planning**

It is recommended that Sea Grant conduct two types of strategic
planning---thematic and programmatic planning. Thematic planning
captures the broad national and state scientific needs and develops a
stakeholder process to prioritize them; while programmatic planning
seeks to marshal and apply Sea Grant\'s research, education, outreach,
legal, administrative and communications capabilities toward reaching
strategic goals.

## D. Identification of Consensus Focus Areas, Goals and Priorities

Once all sources of information are compiled, the university-based
program must identify the top thematic and functional focus areas to
shape the university-based plan. These focus areas must be in alignment
with the national plan and other relevant plans, but will also likely
include topics of specific mandate or interest to the state program that
fit within the needs of the state. It is critical university-based
programs have the flexibility to choose focus areas strategically,
taking into account the variety of funding sources and interests served
under the Sea Grant umbrella within the state. In the determination of
the focus areas, consensus should be the aim. It will be critical to
document the specific opportunities and niches available to Sea Grant
and explain the processes followed for conducting gap and risk analyses
in order to identify those focus (niche) areas. When choosing focus
areas the program will keep the following in mind: Sea Grant's
strengths, societal goals articulated locally and nationally, and needs
articulated by relevant stakeholders. Once focus areas are identified,
they should be shared and discussed with the NSGO Program Officer.

######### **E. Expected Elements of Plans**

Programs may adopt a number of different formats and approaches for
strategic and subsequently, implementation planning. For instance, some
programs may utilize integrated strategic and implementation plans,
while others may separate these into different documents. Regardless of
the final model employed plans should address both thematic and
functional elements and include the following:

-   **Vision:** defined as a description of the ideal future
    contribution/state of the organization. A consciously created image
    of what the organization would ideally like to be;

-   **Goals:** defined as broad strategic positions or conditions the
    organization desires to reach. Goals close the gap between the
    organization\'s preferred vision and its current situation. An
    example are those thematically expressed in the Ocean Research
    Priorities Plan (ORPP) which represent national needs appropriate to
    Sea Grant's niche; that are sufficiently challenging; and where Sea
    Grant can have significant ownership;

-   **Priorities:** defined as the finite areas that will be addressed
    given consideration of vision, capabilities, finances, and
    competitive position;

-   **Objectives:** defined as specific implementation guidelines for
    the goals from the strategic planning exercise, each having a
    specific time frame. They are measurable statements of achievement
    that lead to the accomplishment of a goal. In the short-term and
    through the long-term, they provide indicators of progress;

-   **Activities or Milestones with clearly articulated timelines and
    resources:** defined as statements of how the organization deploys
    resources to accomplish specific objectives. They include specific
    quantifiable activities that can be measured;

-   **Benchmarks:** defined as the tangible markers of success that can
    be quantified by a baseline (current) and target (projected); and,

-   **Expected outcomes:** defined as the specific end result that the
    objectives will meet.

F. Development of Robust Objectives and Milestones for Each Priority and
Incorporation into the Draft Plan

Critical to the success of any program, are tangible, quantifiable
measures of progress and performance. Objectives and milestones should
be incorporated into the university-based strategic/implementation
plan(s).

When developing quantifiable measures:

i)  Include a strategy for allocating resources;

ii) Identify milestones and expected outcomes for the implementation of
    program goals and objectives for the two-year period;

iii) Identify program elements and their context, as well as personnel
     needed;

iv) Highlight the necessary time frame for implementation;

v)  Describe the evaluation process and how you will measure success;
    and,

vi) Identify the degree of interaction and integration with other
    programs (both outside and inside the Sea Grant network).

Metrics should:

i)  Accurately describe and measure relevant aspects of the Sea Grant
    enterprise and be useful to managers at the state and national
    levels;

ii) Take into account the nationally-required performance measures;

iii) Be robust and designed in a manner that allows for broader
     analysis---in particular, they should enable reasonable comparisons
     to be made for individual programs over time, and between
     appropriate program peers;

iv) Be readily attainable---specifically reflect ongoing "internal"
    self-assessment efforts and accumulate continuously; and,

v)  Be evaluated periodically to determine their effectiveness.

######## Appendix III 

######## 

######## Duties of a Federal Program Officer

[(1) Program oversight duties]{.underline}:

-   Review and approve programs\' competitive project selection
    processes.

-   Review and approve omnibus and other program grant applications.

-   Review and approve Annual Reports.

-   Observe program activities, such as interactions with Advisory
    Boards and constituents.

-   Report on, or otherwise address, program grant performance issues.

[(2) Grants management]{.underline}

-   Take mandatory grants Program Officer training.

-   Process omnibus applications and other grant actions.

-   Assure NEPA, budgetary and other required analyses of grant
    applications takes place.

-   Communicate with programs regarding grants policies.

[(3) Program planning]{.underline}

-   Participate in program strategic and annual planning processes.

-   Bring the program perspective into the NSGO annual planning process.

-   Assist programs with implementing program improvement.

[(4) Program assistance and facilitation]{.underline}

-   Facilitate TATs

-   Facilitate NSGCP Director's visit to the university-based Sea Grant
    programs.

-   Participate in NOAA programs whose missions make them logical
    partners with Sea Grant programs.

-   Facilitate communication and collaboration between Sea Grant and
    NOAA programs.

-   When appropriate, represent NOAA at Sea Grant program events.

**Appendix IV**

Strategic Planning Alignment Memo Guidance

Each Sea Grant program will have time to examine the new national plan
and to fully consider how its current programming and existing strategic
plan reflects (or does not reflect) new national plan thrusts. Close
interaction (e.g., telephone, postal and electronic mail, visits, etc.)
between the university-based program's director and assigned NSGO
program officer will facilitate strong alignment, linking the state and
national plans, and acceptance by the NSGO. The strategic planning
alignment memo should be sent directly to the NSGCP Director with copies
sent to the program officer. The memo should not exceed five pages in
length and will be due by April 1, 2008, or as soon as programs have had
the opportunity to conduct processes for staff and stakeholder input.

Critical components of this memo include:

-   Term of the current strategic plan;

-   Identification of goals, objectives and themes for which there is
    already alignment between the university-based program plan and the
    national plan;

-   Description of how elements of the existing plan might be modified
    in order to better align with the new national plan, and how these
    elements will contribute to the national effort to achieve those
    goals and objectives;

-   Description of new elements of the state plan added as a result of
    consideration of the new national plan, and through input from
    stakeholder groups; and,

-   Description of any stakeholder or advisory committee reviews that
    may have been utilized in formulating the alignment memo.

**Alignment memo acceptance (2008)**

Discussions between the PO and university-based director may result in
revisions to the alignment memo. The agreed-upon alignment memo will be
transmitted to the NSGCP Director for final approval and signature.
Approval will be based upon the inclusion of the critical components
listed above, the demonstration of stakeholder involvement, and
demonstration that the plan links directly to elements of the National
Sea Grant Strategic Plan.

**Appendix V**

Strategic Plan Approval by the NSGO (2013)

Starting 2013, strategic plans will be approved by the NSGO.
Conversations between the NSGO program officer (PO) and the
university-based director should be ongoing through the strategic
planning process. Once the final university-based plan is complete, it
should be submitted to the NSGO program officer for acceptance and
approval by the NSGCP Director before adoption.

Steps:

1\) NSGO Program officers will be trained in different strategic
planning models through seminars, discussions and readings.

# 2) The university-based strategic plan should address the clear expectations outlined in **Appendix II,** "Considerations for Developing the Content of Strategic and Implementation Plans (2013)."

3\) The NSGO PO should be involved continuously with the strategic
planning process.

4\) The university-based program will submit the strategic plan to the
NSGO.

5\) The NSGO PO will work in conjunction with the NSGO program planning
and assessment group (PPA) to review the plan. Support for PPA could
include experts from the field of strategic planning.

6\) After input from the NSGO PO and PPA, the NSGCP Director will have
the final approval and signature of each university-based strategic
plan.

> **The NSGO PO and PPA input to the NSGO Director will include a review
> of the following:**

-   The establishment of a planning process must be evident.

-   The planning process must demonstrate the appropriate involvement
    and endorsement of constituency and advisory groups (i.e. Advisory
    Board, university representatives and others) at every level; this
    involvement and endorsement should include conferring with
    stakeholders prior to the drafting and development of the plan, and
    continuing through plan approval, implementation, monitoring and
    evaluation.

-   Evidence must be presented that shows that the plan is subject to
    frequent review.

-   The plan reflects local, state and/or regional needs.

-   The plan demonstrates ties to the NSGO and NOAA Strategic Plans.

-   The plan's priorities and selection process are clearly articulated.

-   The plan at least contains and specifically addresses the following
    core elements: vision, goals, objectives, priorities, benchmarks and
    expected outcomes.

-   The plan is not so rigid as to preclude responding to issues and
    opportunities as they arise.

-   Planning is done with other institutional and agency resources in
    mind, and complementary or supplementary programs are planned as
    appropriate.

-   The plan includes both short and long-term programmatic and
    management goals.

-   The plan demonstrates links from state to regional to national
    priorities.

**Appendix VI**

Topical Assistance Team (TAT) Guidance

The purpose of these guidelines is to establish a framework for TAT
procedures and to provide guidance to all parties involved with
conducting TAT reviews (programs, NSGO, NSGRP, institution
administrators, faculty, staff and stakeholders).

In 1998, the National Sea Grant Office (NSGO) issued a policy document
on the implementation of program assessment procedures in the National
Sea Grant College Program. The policy document emphasized that program
improvement and assessment "must be an ongoing, continuous process." The
document highlighted a new procedure for on-going self-improvement and
assessment termed a topical assistance team (TAT), which provided an
opportunity for a program to receive outside advice by a small team of
experts on a specific program element and/or a specific program
management mechanism. Many programs have utilized TAT procedures over
the past decade to improve their program.

All programs have elements that can be improved, or new management
directions that can be explored. Under the new planning and assessment
system, utilizing a TAT is mandatory once each five-year review period.
The purpose of a TAT is to provide the opportunity for a small team of
persons knowledgeable in the topical area, working with the program
director and invitees, to evaluate and offer advice on an element or
area of the Sea Grant program through an exchange of information. The
team's responsibility is to render opinions, options and conclusions in
a written report in an effort to improve the program element under
review. TATs provide a focused, intensive review of a program element
(management, research sub­program, communication, education, extension)
or a specific issue that needs to be addressed. Examples might include
implementing a new program planning process, developing a new research
sub-program or re-organizing the communications program to utilize
emerging communications technology. Because assessment decisions are
de-coupled from the process, TAT reviews provide a venue for open and
frank dialogue among the TAT members and affected parties in the
program. TATs provide external expert opinion which can often be helpful
to move internal, institutional decisions. The focus of a TAT should be
decided upon by the director in consultation with the program officer.

**Topical Assistance Team Review Procedures**

**[Initiation Process]{.underline} --** A TAT review must be conducted
at least once during the five-year review cycle. Both the program
director and the NSGO PO must mutually agree on the topic of the review,
which may be informed by a number of sources, including but not limited
to the PRP assessment. The director/PO agreement should also address how
the costs of the topical assistance team review will be assigned. In
general, the costs of TATs should be shared, with the NSGO covering
costs and expenses associated with organizing and conducting the review
team, and the program covering the costs associated with participation
by faculty, staff, stakeholders, etc, at the program level. However,
funding flexibility should be maintained to allow for unforeseen
budgetary scenarios which should be negotiated between the PO and
director. For planning, scheduling and budgeting purposes, parties are
encouraged to request TAT reviews by October 1 of the fiscal year in
which the assessments are intended to take place. Given multiple program
responsibilities by POs, the NSGO may have to negotiate an overall
schedule with the directors to distribute TATs evenly over the five-year
review period.

**[Scope of Advisory Team Reviews]{.underline} --** Once a TAT review
has been agreed to in principle, the scope of the review, the review
objectives, and a tentative agenda will be determined by the director of
the Sea Grant program in consultation with the NSGO PO. The Sea Grant
director is responsible for communicating the purpose, procedures and
schedule of the review to appropriate senior university officials,
faculty, staff or stakeholders. In general, the senior university
officials are encouraged to attend the exit interview and to participate
in the advisory team review as appropriate. Because of their particular
expertise, TAT members should be provided an opportunity for input to
the agenda once the TAT is established.

**[Roles and Responsibilities for TAT Leadership]{.underline} --**

*NSGO Program Officer --* The PO is responsible for agreeing on behalf
of the NSGO to conduct the review on a given topic and time. The PO is
also responsible for formally appointing members of the TAT after
receiving recommendations for TAT members from the Sea Grant director.
The PO, after consulting with the Sea Grant director, will name a chair
from among the TAT members. A TAT typically has three to five members.
The PO will participate as an ex-officio member of the TAT, serve as
staff to it, and assume responsibilities for compiling, editing and
distributing the final TAT report.

*Sea Grant Director --* The Sea Grant director of the host program is
responsible for agreeing on behalf of the program to host the review.
The Sea Grant director is also responsible for working with the PO to
establish the goals, objectives and scope of the review and for
logistics and travel arrangements within state during the review.

*NSGRP --* Depending upon the expertise needed for the TAT, it is
expected that at least one member of the Panel will participate in a
TAT. The PO will coordinate Panel member participation with the NSGO
Deputy Director and the Chair of the Panel on an annual basis. Panel
members will be selected for TAT participation based on their particular
expertise.

**[TAT Report]{.underline} --** A written report and an exit interview
are required as part of the TAT process. The exit interview will be
conducted by the TAT Chair prior to the team leaving the institution.
Although it is expected that all TAT members will contribute sections of
the written report, it is the responsibility of the NSGO PO to compile
the final draft, which will be signed by the Chair of the TAT and
distributed to the program from the NSGO. Typically, the TAT report will
be distributed to the Sea Grant director and his/her institutional
supervisor unless arrangements are negotiated for other distribution
during the preliminary request for a TAT. Because federal funds are used
to conduct TATs, TAT reports will be entered into the permanent NSGO
program file. In order to a maintain separation between program
self-improvement and program assessment processes, the TAT report will
only be made available to the PRP or PRP-sub upon the request of the
director.

# **Appendix VII**

# 

# 

# Program Review Panel

The Program Review Panel (PRP) is a new structure within the Sea Grant
assessment system. It is designed in part to address concerns regarding
the Program Assessment Team model used in the past two assessment
cycles. Specifically, the PRP will utilize a panel of well qualified
individuals to evaluate all programs based on a consistent set of review
documents (briefing book, annual reports and strategic plans) as well as
direct interactions with program directors. The review will take place
in the fifth year of the assessment cycle (see **Table I**, page 14) and
will provide numerical scores for each program. The NSGCP Director will
utilize these to assign a ranking that will set merit and bonus fund
allocations as appropriate. The PRP will also establish the initial
program ratings that will be the baseline for the new assessment
process. In addition, a subcommittee of the PRP will be responsible for
the mid-course review (see **Appendix VIII,** page 45).

**PRP Composition**

The PRP will be appointed by the NSGCP Director who may seek advice on
and nominations for members as he/she sees fit. The PRP will be
comprised of fifteen approximately (15) individuals encompassing a wide
range of expertise. Research, outreach and education disciplines must be
included on the panel, and all members should have a direct knowledge of
and appreciation for Sea Grant in concept and practice. Each panel
member must develop a strong working knowledge of the review process,
its performance metrics and links to strategic planning.

**Expectations of each PRP Member**

The PRP members must be willing to contribute a minimum of 4-6 weeks of
time in the review year. A sub-set of the PRP (five or six members) will
be expected to participate in the mid-course review. It is anticipated
that members will be compensated fully at appropriate federal rates for
their participation on the PRP. PRP Meetings may take place at NOAA
headquarters in Silver Spring or at appropriate locations nationwide as
determined by the NSGCP Director. Each PRP member must read all briefing
materials for every program and be prepared to discuss and score them.
More detailed analyses by members of the PRP are expected for a subset
of programs under review (see below).

**Organization of the PRP Review**

The 31 Sea Grant programs will be divided into two groups to be
evaluated in one of two weeklong sessions. These sessions will be
separated by a one-week "break" for the PRP. Each panel member will be
expected to review briefing materials for every program and be prepared
to discuss them in plenary session. In addition, each panelist will be
assigned as the primary, secondary or tertiary reviewer for a set of
programs. The primary reviewer will be responsible for leading the
discussion on each program with substantive input from secondary and
tertiary panelists. Written briefing materials are central to the
process (see below). In addition, each Sea Grant director (or designee)
will have the opportunity to answer questions. This "Director's Q&A"
will be facilitated by the primary panelist and will also include at
least one of the secondary or tertiary reviewers. A written summary of
the session will be prepared. It is anticipated that these sessions will
take about 1-2 hours. The POs will be available to answer any questions
during the discussion of their respective programs.

A possible chronology of the PRP review would include the following:

Days 1-2: PRP members discuss all programs individually in plenary
session for the first group of 15 or 16 programs under review.
Discussions will be lead by the primary panelist with input from the
secondary and tertiary reviewers. Questions and comments can be provided
by these reviewers and/or any other member of the PRP. Points of
clarification and questions will be annotated by the primary panelist
and brought to the attention of the Sea Grant director.

Day 3: Each Sea Grant director will meet with their assigned panelists
for approximately 1-2 hours. While in-person meetings are encouraged,
conference calls are also appropriate.

Days 4-5: The full PRP will re-convene to discuss each program in a
format that follows the discussions held on days 1-2. The primary
panelist will convey the results of the Director's Q&A to the PRP at
this time. Following this discussion, each PRP member will provide a
score based on approved performance metrics. The primary panelist will
be responsible for summarizing PRP discussions as well as additional
information gleaned from discussions with the director. This PRP summary
will be provided to the NSGCP Director.

The process will be repeated for the remaining programs after a hiatus
of one week.

**The Role of the NSGO Post-PRP**

Upon the completion of the entire review process, all scores will be
provided to the NSGCP Director who will collate them and generate a
final rating for each program. The summary rating and written PRP
Summary will be conveyed to each program director. Ratings will stand
for a minimum until the mid-course review.

**Briefing Materials**

Each program will be required to submit a package of materials to the
PRP. Included, will be a briefing book analogous to those prepared in
previous assessment cycles. A greater emphasis will be placed on the
link between planning and outcomes for this review. However within the
confines of a standard format, (i.e., page length and required
components) programs will be able to describe how they have responded to
state, regional and national priorities as well as emerging
opportunities and unanticipated needs.

**Performance Metrics and Scoring Criteria**

The PRP will score programs based on a set of criteria developed once
strategic priorities are established for the National Sea Grant Program.
It is anticipated that the NSGCP Director will establish a task force to
accomplish this in late 2007 or early 2008. Consideration should be
given to using appropriate metrics from the previous assessment system
with emphasis placed on outputs of quality and quantity that contribute
to planned goals. Consultation with experts in the development of
performance metrics for science-based programs is anticipated as the
task force develops its guidance.

###### Appendix VIII

###### 

###### Mid-Course Review *(2011)* The proposed midcourse review system will provide an opportunity for program ratings to be adjusted either up or down for demonstrated cause midway through the five-year review period. The midcourse review encourages self-improvement by providing an opportunity for the program to receive higher ratings for improvements made in response to suggestions by the PRP. 

###### Team composition and charge A mid-course review sub-group, comprised of five or six members from the original 15-member PRP, will be convened in 2011, two years after the initial rating review. This sub-group of the PRP (PRP-sub) will be charged with reviewing new information provided since the last PRP review and advising the NSGCP Director on changes that should be made to the original 2009 program ratings. The PRP-sub will be familiar with and have available the rationale provided for the previous rating, the two most recent annual reports, and a formal request from the state program director for a rating change based on good cause. The rating criteria for this 2011 mid-course review will remain the same as for 2009 initial rating review, but will change to the new performance criteria in subsequent reviews. 

###### Rating changes* *It is expected that the formal request from the state program director for a rating change will be concise, (a short memo of no more than two or three pages) and directed at one or more of the reasons provided by the PRP for a less than optimal rating. Alternately, for justified cause, a state program's PO, or the NSGCP Director, based on observations from the on-site visit or other evidence, can request a lower rating by the PRP-sub for one or more rating criteria. A request for a lower rating must be made transparent, and presented in writing, with copies of the request provided to the state program director, who will have an opportunity to respond in writing. 

###### The PRP-sub is not a decision-making entity, but rather is an independent panel which advises the NSGCP Director on whether or not there is enough evidence to justify a rating change for any program. The NSGCP Director, in consultation with the state program's PO, will make all final decisions regarding a rating change. It is anticipated that these mid-course review procedures will encourage and reward program self-improvement and provide each program an opportunity to petition against what the program perceives to be an unfair rating. Similarly, if a program is determined to be regressing in a rating category, there is an opportunity to reflect the poorer performance through a revised rating. 

###### For the 2011 mid-course review, Sea Grant will still be in a transition period as it moves toward full implementation of the new planning and assessment system. The 2009 PRP ratings, which will be the subject of review and possible adjustment in 2011, will be based on the previous criteria of the former program assessment system. The rating criteria for the 2011 mid-course review will therefore rely on the original program assessment criteria as described in detail in the Program Assessment Team Manual and as applied by the 2009 PRP. In addition, the state program will have specific guidance and criteria on how the program can improve from the original program ratings by the PRP. Further, the PO and the state program director should be in regular communication about steps underway toward program improvement. 

###### The mid-course review is ideally suited for acknowledging and rewarding program self improvement. Over time, the ultimate goal is to have all Sea Grant programs rate in the highest performance category. It is important to note, however, that under the current ranking scheme required by Sea Grant legislation, re-rating one program higher into one of the top two rating categories would necessarily require moving another program out of that same category and into a lower one. If this congressionally mandated ranking system still remains in the 2008 Sea Grant legislation, then all programs, including those with highest ratings, may be affected by the revised ratings of some. Therefore, all programs will be provided the opportunity to request a rating change, or to provide rationale for maintaining their current high rating, by summarizing additional improvements or new outcomes that have occurred in their program. The annual reports should contain the relevant new information from which the program makes it case to maintain a rating or to be rated in a higher category. 

**Appendix VIV**

RIT Response to Individual NRC Report Recommendations

**The RIT Response to the Individual Recommendations of the National
Research Council Report: Evaluation of the Sea Grant Program Review
Process, June 16, 2006; National Research Council**

**Recommendation 1: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce and in
consultation with the National Sea Grant Review Panel and the individual
Sea Grant programs, should strengthen the ability of the National Sea
Grant Office to carry out meaningful, ongoing internal assessment in
order to complement periodic, external assessment currently taking
place.**

**Location of Recommendation 1:** Summary Section/Effectiveness
Post-2002 Evaluation (p. 5); and Chapter 5/Major Findings &
Recommendations/Effectiveness of Post-1998 Evaluation (p. 89)

**RIT Response to Recommendation 1:** By implementing several new
components of the planning and assessment system outlined in this
report, program officers will be more routinely involved with and
informed of Sea Grant program operations. Thus, implementing ongoing
periodic assessments will be more easily achievable. These new
components include: 1) involving the NSGO POs in the state Sea Grant
program's strategic planning process, including the plan's approval; 2)
on-site visits by the Director of the NSGCP; and, 3) TATs. In addition,
we recommend that the Director make as a priority the support and
funding of new program officers, and be prepared to make necessary
trade-offs by limiting, as needed, other non-PO activities.

**Recommendation 2: Steps should be taken by the Director of the
National Sea Grant College Program, under supervision of the Secretary
of Commerce and in consultation with the National Sea Grant Review Panel
and the individual Sea Grant programs, to strengthen strategic planning
at both the national and individual program level. The strategic plans
of the individual programs and the national program should represent a
coordinated and collective effort to serve local, regional, and national
needs.**

**Location of Recommendation 2:** Summary Section/Strategic Planning (p.
6); and Chapter 5/Major Findings & Recommendations/Strategic Planning
(p. 91)

**RIT Response to Recommendation 2:** We agree with this recommendation
and its implementation is well underway.

**Recommendation 3: Each individual Sea Grant program, in collaboration
with its local network and the National Sea Grant Office, should develop
an appropriately ambitious, high quality strategic plan that meets local
and institutional needs while simultaneously reflecting the individual
program's role in addressing the regional and national needs identified
in the strategic plans of NOAA and National Sea Grant College Program.**

**Location of Recommendation 3:** Chapter 4/Program Oversight and
Management/Findings and Recommendations Regarding Program Oversight and
Management/Strategic Planning Process (p. 84)

**RIT Response to Recommendation 3:** We agree with this recommendation.
The foundation of the new planning and evaluation system proposed by the
RIT is inherently dependent on ambitious, high quality state strategic
plans. Throughout the RIT report, the need for both a top down and
bottom-up approach to strategic planning is emphasized.

**Recommendation 4: The National Sea Grant Office, in consultation with
the National Sea Grant Review Panel and individual Sea Grant programs,
should establish regular procedures (separate from annual and periodic
performance evaluation) for working with the individual Sea Grant
program to create and adopt an appropriately ambitious strategic plan,
with goals and objectives against which the program would be evaluated
at the next program evaluation period.**

**Location of Recommendation 4:** Chapter 3/Critique of the Periodic
Assessment Process/Findings and Recommendations Regarding the Periodic
Assessment Process (p. 67)

**RIT Response to Recommendation 4:** We agree with this recommendation.
As per the advice of a strategic planning consultant, the goals and
objectives against with which individual Sea Grant programs will be
evaluated will be developed in November by a separate committee once the
national strategic plan is developed. A consultant will be employed to
assist in the development of benchmarks and performance measures that
emanate from the goals and objectives of the new national Sea Grant
strategic plan.

**Recommendation 5: The Director of the National Sea Grant College
Program, in consultation with National Sea Grant Review Panel, should
formally review and approve each individual strategic plan.**

**Location of Recommendation 5:** Chapter 4/Program Oversight and
Management/Findings & Recommendations Regarding Program Oversight and
Management/Strategic Planning Process (p. 84)

**RIT Response to Recommendation 5:** The Director of the National Sea
Grant College Program, in consultation with the individual Sea Grant
program's PO, will formally review and approve each individual strategic
plan. We do not agree that the National Sea Grant Review Panel needs to
consult at the level of review and approval of each program's strategic
plan. Rather, we believe the Panel should concentrate its efforts on
participating in and reviewing the National Sea Grant Strategic Plan.

**Recommendation 6: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce and in
consultation with National Sea Grant Review Panel and the individual Sea
Grant Programs, should modify the benchmarks and indicators, as needed,
to ensure that the performance of each program is measured against the
objectives outlined in the separately approved, program specific
strategic plan called for in the previous recommendation.**

**Location of Recommendation 6:** Summary Section/Performance Criteria
(p. 6); and Chapter 5/Major Findings & Recommendations/Performance
Criteria (p. 92)

**RIT Response to Recommendation 6:** See response to recommendation
Number 4**.**

**Recommendation 7: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce and in
consultation with National Sea Grant Review Panel and the individual Sea
Grant programs, should substantially reduce the overall number of scored
criteria by combining various existing criteria, while adding
cooperative, network-building activities as an explicitly evaluated,
highly valued criterion.**

**Location of Recommendation 7:** Summary Section/Performance Criteria
(p. 7); and Chapter 3/Critique of the Periodic Assessment
Process/Findings and Recommendations Regarding the Periodic Assessment
Process (p. 65); and Chapter 5/Major Findings &
Recommendations/Performance Criteria (p. 92)

**RIT Response to Recommendation 7:** We agree with this recommendation.
The details of the scoring system will be developed once the focus areas
of the national strategic plan are known. However, it is our intent that
the overall number of scored criteria be reduced and that a scoring
incentive be introduced to encourage and reward Sea Grant programs for
participating in network building activities.

**Recommendation 8: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce, should revise
the calculation of bonus funding allocation relative to program rank to
ensure that small differences in program rank do not result in large
differences in bonus funding, while preserving or even enhancing the
ability to competitively award bonus funds as required by the National
Sea Grant College Program Act Amendments of 2002 (P.L. 107--299).**

**Location of Recommendation 8:** Summary Section/Fairness in
Competition (p. 9); and Chapter 3/Critique of the Periodic Assessment
Process/Findings and Recommendations Regarding the Periodic Assessment
Process (p. 67), and Chapter 5/Major Findings & Recommendations/Fairness
in Competition (p. 95)

**RIT Response to Recommendation 8:** We agree with this recommendation.

**Recommendation 9: The Director of the National Sea Grant College
Program, working with the National Sea Grant Review Panel, should
carefully review the present benchmarks and indicators to ensure that
they are sufficiently ambitious and reflect characteristics deemed of
high priority for the program as a whole.**

**Location of Recommendation 9:** Chapter 3/Critique of the Periodic
Assessment Process/Findings and Recommendations Regarding the Periodic
Assessment Process (p. 64)

**RIT Response to Recommendation 9:** See response to recommendation
Number 4.

**Recommendation 10: The Director of the National Sea Grant College
Program, working with the National Sea Grant Review Panel, should engage
independent expertise to refine the benchmarks and grading instructions
to meet professional methods and standards for reliability and to refine
the training materials used to prepare individuals involved in the
evaluation process, in a manner consistent with the recommendations made
in this report.**

**Location of Recommendation 10:** Chapter 3/Critique of the Periodic
Assessment Process/ Findings and Recommendations Regarding the Periodic
Assessment Process (p. 65)

**RIT Response to Recommendation 10:** An independent consultant has
been retained and will assist the NSGO to refine benchmarks and grading
instructions once the national strategic plan is completed.

**Recommendation 11: The Director of the National Sea Grant College
Program should communicate the results of the FE (annual NSGO Final
Evaluation) directly to individual Sea Grant program directors. This
communication should include the final rating score received by that
program (as begun in 2004) and document any substantial difference
between the conclusions reached during the annual evaluation and the
most recent periodic review. Furthermore, the Director of the National
Sea Grant College Program should communicate the implication of the
annual evaluation in terms of the rating and ranking process used to
determine a program's eligibility or receipt of merit or bonus
funding.**

**Location of Recommendation 11:** Chapter 3/Critique of the Periodic
Assessment Process/Findings and Recommendations Regarding the Periodic
Assessment Process (p. 66)

**RIT Response to Recommendation 11**: We agree with the intent of this
recommendation. However, in our proposed system, Sea Grant programs will
be rated once every five years, and this rating will be reviewed midway
during the rating period at what is called the mid-course review.

**Recommendation 12: The Director of the National Sea Grant College
Program, in consultation with the National Sea Grant Review Panel,
should work to establish an independent body to carry out the periodic
assessments under the supervision of the National Sea Grant Review
Panel.**

**Location of Recommendation 12:** Chapter 4/Program Oversight and
Management/Annual and Periodic Assessment Processes as Integral Elements
of Program Administration (p. 79)

**RIT Response to Recommendation 12:** A key component of our proposed
system is the introduction of the Program Review Panel, an independent
body of experts appointed for the purpose of carrying out periodic
(five-year) and mid-course reviews, which will result in individual
state program ratings. Although we believe the Director of the National
Sea Grant College Program should consult with the National Sea Grant
Review Panel, we believe that the Director, and not the Panel, should
supervise the implementation of periodic assessments and the management
of the Program Review Panel.

**Recommendation 13: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce and in
consultation with National Sea Grant Review Panel and the individual Sea
Grant programs, should shorten the duration and standardize the PAT site
visits, based on the minimum time and material needed to cover
essential, standardized elements of the program assessment.**

**Location of Recommendation 13:** Summary Section/Program Assessment
Teams and Site Visits (p. 7); and Chapter 5/Major Findings &
Recommendations/Program Assessment Team and Site Visit (p. 93)

**RIT Response to Recommendation 13:** The RIT's proposed integrated
planning and assessment system does not utilize PAT site visits. PAT
visits are replaced by on-site visits by the Director of the National
Sea Grant College Program and TATs. These Sea Grant program visits will
be more focused, and will reduce the time and costs required to host
visits by the programs.

**Recommendation 14: National Sea Grant Office and National Sea Grant
Review Panel should reduce the effort and costs required to prepare for
and conduct a Program Assessment Team site review by providing specific
limits on the amount and kind of preparatory material to be provided to
the Program Assessment Team and by limiting the site visit to no more
than three days, including the time to draft the preliminary report and
meet with program directors and institutional representatives.**

**Location of Recommendation 14:** Chapter 3/Critique of the Periodic
Assessment Process/Findings and Recommendations Regarding the Periodic
Assessment Process (p. 66)

**RIT Response to Recommendation 14:** See response to recommendation
Number 13.

**Recommendation 15: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce, should rank the
individual Sea Grant programs based on a program evaluation process that
includes more robust, credible, and transparent annual assessments of
each individual Sea Grant program.**

**Location of Recommendation 15:** Summary Section/Providing
Coordination and Facilitation Through Informed, Ongoing Oversight (p.
8); and Chapter 5/Major Findings and Recommendations/Providing
Coordination and Facilitation Through Informed, Ongoing Oversight (p.
94)

###### RIT Response to Recommendation 15: The NRC emphasized the need for annual assessments, in part to recognize and reward programs for improvements made on a more regular basis, and in part to de-emphasize the event nature of the previous program assessment system. The RIT concluded that wholesale rating reviews of the programs on an annual basis would be burdensome to all parts of the network. In our system, accountability and outcomes will be tracked annually through annual reports. We have proposed a mid-course review system through which a program's rating can be adjusted either up or down for demonstrated cause. 

**Recommendation 16: The Secretary of Commerce, in consultation with the
National Sea Grant Review Panel, should take steps to ensure that
sufficient human and fiscal resources are available to allow robust,
ongoing, and meaningful interaction among the Director of the National
Sea Grant College Program, the staff of the National Sea Grant Office,
the directors of individual Sea Grant programs, and the administrators
of the home institutions of individual Sea Grant programs.**

**Location of Recommendation 16:** Summary Section/Improving Program
Cohesion (p. 9); and Chapter 5/Major Findings &
Recommendations/Improving Program Cohesion (p. 96)

**RIT Response to Recommendation 16:** We agree with this
recommendation.

**Recommendation 17: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce and in
consultation with National Sea Grant Review Panel and the individual Sea
Grant programs, should under-take an evaluation of how work force
capabilities and other components of effective program administration
could be modified within the National Sea Grant Office to enhance its
ability to coordinate and facilitate the actions of the individual Sea
Grant programs.**

**Location of Recommendation 17:** Summary Section/Improving Program
Cohesion (p. 9-10); and Chapter 5/Major Findings and
Recommendations/Improving Program Cohesion (p. 96)

**RIT Response to Recommendation 17:** The National Sea Grant Office
Operations Sub-Team addressed the issue of the need for additional
resources that would be necessary to implement the RIT proposal. Once a
new system is endorsed by the Director of the National Sea Grant College
Program, we agree that the Secretary of Commerce should look at new work
force capabilities, with an emphasis on what other NSGO administrative
duties could be modified or eliminated.

**Recommendation 18: The Director of the National Sea Grant College
Program, under supervision of the Secretary of Commerce, should take
steps to ensure that the program assessment process (both the new annual
assessment called for in this report and the Program Assessment Team
review) is well-described and understood by individual program
directors, congressional staff, personnel at the Office of Management
and Budget, university and state administrators, and the general
public.**

**Location of Recommendation 18:** Summary Section/Improving Program
Cohesion (p. 10); and Chapter 5/Major Findings and
Recommendations/Improving Program Cohesion (p. 97)

**RIT Response to Recommendation 18:** We agree with this
recommendation.

**Recommendation 19: The Director of the National Sea Grant College
Program, acting under authority of the Secretary, should direct the
National Sea Grant Review Panel to undertake the development of a
systematic review of the "state of the Sea Grant program" once every
four years. The review should rely extensively on information collected
during the annual and periodic assessments, augmented with a site visit
to the National Sea Grant Office, and it should focus on how the program
is functioning as a whole.**

**Location of Recommendation 19:** Summary Section/Improving Program
Cohesion (p. 10); and Chapter 4/Findings and Recommendations Regarding
Program Oversight and Management/Increasing Reliability and Transparency
of Annual and Periodic Assessment (p. 85); and Chapter 5/Major Findings
and Recommendations/Improving Program Cohesion (p 97)

**RIT Response to Recommendation 19:** We agree, but recommend the
"State of the Sea Grant Report" be developed every two years.

**Recommendation 20: The Director of the National Sea Grant College
Program should ensure that program administration carried out by the
National Sea Grant Office makes full and consistent use of annual
reporting, frequent and meaningful interactions with individual Sea
Grant programs by National Sea Grant Office program officers, and the
development, approval, and implementation of strategic plans to monitor
and assess the performance of the individual Sea Grant programs on an
ongoing basis.**

**Location of Recommendation 20:** Chapter 4/Findings and
Recommendations Regarding Program Oversight and Management/Sea Grant
Program Administration (p. 82)

**RIT Response to Recommendation 20:** We agree with this
recommendation.

**Recommendation 21: The Director of the National Sea Grant College
Program, working with the National Sea Grant Review Panel, should
redirect the focus from periodic external Program Assessment Team
reviews towards identifying areas and mechanisms for improving the
individual Sea Grant programs as well as the National Sea Grant Office's
efforts to facilitate and coordinate program efforts.**

**Location of Recommendation 21:** Chapter 4/Findings and
Recommendations Regarding Program Oversight and Management/Sea Grant
Program Administration (p. 83)

**RIT Response to Recommendation 21:** This recommendation is central to
the approach the RIT undertook in developing the planning and assessment
system outlined in our report. Evaluation has been redirected away from
PAT-related site visits to a Program Review Panel comprised of
independent experts. Visits to the programs will include on-site visits
by the NSGCP Director, and mandatory TATs, both of which will be
conducted for the purpose of individual Sea Grant program
self-improvement.

**Recommendation 22: The Director of the National Sea Grant College
Program, in consultation with the National Sea Grant Review Panel,
should create a process for determining the underlying causes of
disagreement for instances where a Program Assessment Team review
appears to reach conclusions at odds with the most recent annual
assessment provided by the National Sea Grant Office.**

**Location of Recommendation 22:** Chapter 4/Findings and
Recommendations Regarding Program Oversight and Management/Sea Grant
Program Administration (p. 83)

**RIT Response to Recommendation 22:** This recommendation is not
applicable in our proposed system.

**Recommendation 23: In order to effectively administer the Sea Grant
program, the Director of the National Sea Grant College Program should
take steps to ensure that sufficient qualified staff are available to
interact with the individual Sea Grant programs, to ensure effective
two-way communication, and to monitor and assess program performance on
an ongoing basis.**

**Location of Recommendation 23:** Chapter 4/Findings and
Recommendations Regarding Program Oversight and Management/Role of the
National Sea Grant Office (p. 83)

**RIT Response to Recommendation 23**: We agree with this
recommendation.

**Recommendation 24: The Director of the National Sea Grant College
Program, in consultation with the National Sea Grant Review Panel and
the directors of the individual Sea Grant programs, should modify the
NSGO Final Evaluation review process so that every individual Sea Grant
program is rated and ranked each year. The rating (and subsequent
ranking) should be based on an assessment of each program's progress for
the reporting year based on annual reports of activities, outcomes, and
impacts in the context of the unique strategic plans approved for each
program.**

**Location of Recommendation 24:** Chapter 4/Findings and
Recommendations Regarding Program Oversight and Management/Increasing
Reliability and Transparency of Annual & Periodic Assessment (p. 85)

**RIT Response to Recommendation 24:** We agree. However, we propose a
mid-course review that will provide the opportunity for a program to
have a rating adjusted once during the five-year rating cycle. See
response to recommendation Number 15.