Review of the Draft Performance Standards and

Indicators for Artificial Production in the

Northwest Power Planning Council’s

Artificial Production Review

Independent Scientific Advisory Board

Northwest Power Planning Council

National Marine Fisheries Service

851 S.W. 6th Avenue, Suite 1100

Portland, OR 97204

Peter A. Bisson

Charles C. Coutant

Daniel Goodman

Robert Gramling

James Lichatowich

Eric Loudenslager

William Liss

Lyman McDonald

David Philipp

Brian Riddell
February 23, 2000

ISAB 2000-2
ISAB 2000-2: Review of APR Performance Standards and Indicators

1

ISAB Review of the Performance Standards and Indicators for Artificial

Production in the Council’s Artificial Production Review

In the ISAB’s letter of January 20, 2000 to the Northwest Power Planning Council, we

proposed to address the Council’s request to review the draft Performance Standards and

Indicators appended to the Council’s Artificial Production Review (APR) by preparing

three documents dealing with the following topics:

1. Appropriateness of the Draft Performance Standards and Indicators

2. Scientific Basis for the Evaluation Process

3. Development of an Appropriate Data System

This is the first of these reports, addressing the appropriateness of the hatchery

performance standards and indicators.

Background:

There are currently at least 120 individual hatcheries rearing and releasing juvenile trout

and salmon into the Columbia basin rivers and streams. Collectively, these individual

hatcheries should constitute an artificial production program for the Columbia River

system. There is, however, no existing legal/administrative structure that can implement

and coordinate a basin-wide artificial production program. Reasons for failure to establish

a legal/administrative structure include the following facts: individual facilities have been

constructed or modified over a 60-year period; they were brought into existence through

several different administrative and legal mechanisms; and they are now operated by

federal, state, and tribal entities.

Initially the primary goal of those facilities was to provide juvenile salmonids for stocking

the ocean to support commercial and sport fisheries and to mitigate for lost or degraded

habitat. Most of the propagation activity was concentrated below Bonneville Dam. In the

last three decades, production has been expanded above Bonneville and a conservation

component (i.e., to help rebuild wild populations in river reaches where habitat is

underutilized) added to the intended goals of those programs. Low priority usually given
ISAB 2000-2: Review of APR Performance Standards and Indicators

2

to monitoring and evaluation and logistical difficulties have prevented the integrated

evaluation of the entire artificial propagation program on a basin-wide scale.

In spite of releasing increasing numbers of smolts from hatcheries, the commercial catch,

escapement to Bonneville Dam, and numbers of wild spawners have continued to decline.

Based on that observation, concern has been raised over the efficacy of the artificial

production program to fulfill its intended purpose. Drawing from a growing body of

ecological and genetic theory scientists have proposed that competition, predation,

disease, and genetic/evolutionary fitness problems result from the interaction of hatchery

origin and wild salmonids. Hatchery produced fish are considered one of the factors in

the decline of wild salmonids. The reports of the Independent Scientific Review Panel

(ISRP) on the Council’s 1998 and 1999 Fish and Wildlife Program recommended that

funding for new hatcheries not be approved until a comprehensive review of artificial

production in the basin is completed. In their FY2000 report, the ISRP identified

reservations with supplementation projects, resident fish hatcheries, and captive brood

technology.

Policy-makers followed up the concerns raised by scientists. The Council’s 1995 Fish

and Wildlife Program called for coordination and evaluation of artificial production. A

regional analysis of supplementation programs (RASP) and an integrated hatchery

operations team (IHOT) were developed, but the comprehensive coordination and

evaluation called for in the Fish and Wildlife Program were not completed.

In 1997, Congress directed the Council to develop a coordinated plan for artificial

production in the Columbia basin with the help of the ISAB and cooperation of Federal,

State, and Tribal agencies. This effort produced the Council’s Artificial Production

Review (APR), which was submitted to Congress in November 1999. The Council

produced the review with considerable input from the Production Review Committee,

which included representatives from Federal, State, and Tribal government agencies, and

conservation organizations. The intent was to create regional support for the review by

providing participation for stakeholders throughout the process. In addition, the Council

formed a Scientific Review Team (SRT) that reviewed the science of artificial propagation.

The APR included (but was not limited to) a set of scientific principles from the multispecies framework, a set of policies to guide the coordination of artificial production in
ISAB 2000-2: Review of APR Performance Standards and Indicators

3

the basin, a list of implementation recommendations, and a set of draft performance

standards and indicators. Importantly, the Council indicated that its APR policies were to

be consistent with the scientific framework.

The APR components should be linked logically to one another. Policy is intended to be

consistent with the set of scientific principles, the implementation recommendations are

intended to facilitate implementing policy, and the performance measures are intended to

provide a yardstick to determine if policy is indeed being implemented.

The performance standards and indicators were in draft form in the report to Congress

and the ISAB has been requested to review their adequacy. The ISAB was asked to

address the following three questions:

* Are the draft performance standards and associated indicators the appropriate tools to

periodically evaluate the effects of individual artificial production programs for the

purpose of determining whether the principles, policies, and purposes in the Artificial

Production Report (APR) are being fulfilled?

* Are the draft performance standards and associated indicators the appropriate tools to

adequately evaluate the effects of the artificial production activities in the basin?

* If a performance standard, indicator or other means of measurement is not the most

appropriate tool for this purpose, what other standard or indicator would you

recommend?

Assessment of Scope:

The ISAB believes that any evaluation of the artificial production program for the

Columbia River Basin requires assessments beyond the level of the individual hatchery.

There needs to be an evaluation of how artificial production activities at individual

hatcheries are integrated to meet the objectives and policies set at the subbasin, province,

and basin levels, as well. The province and basin levels of evaluation cannot be

accomplished simply by compiling performance evaluation information collected from a

subset or even all of the hatcheries in the basin.
ISAB 2000-2: Review of APR Performance Standards and Indicators

4

To accomplish that task, objectives must be developed for each of the hierarchical levels,

i.e., hatchery, subbasin, province, and basin. It should be stressed here, however, that the

purpose for each hatchery, and, hence its hatchery-level objectives must be consistent

with subbasin and larger-scale objectives. The different objectives at each level in the

spatial hierarchy will require a different set of performance standards and indicators.

Each set of performance standards and indicators must be designed to assess a different

level of the hierarchy, from very specific tactical questions for individual hatcheries to

quite broad strategic questions for the basin as a whole.

The APR identifies five different purposes for artificial production: augmentation,

mitigation, restoration, preservation/conservation, and research. Because the objectives at

the province and basin levels would likely use artificial production for multiple purposes,

it is necessary to devise a different set of performance standards and indicators for each of

the four hierarchical levels.

Consider this hypothetical example: Assume that the purpose, specific objectives, and

relation to large-scale objectives for a given hatchery have been determined. The

hatchery-level evaluation would use the set of performance standards and indicators

designed to determine if the purpose and specific objectives of the hatchery are being

achieved. All hatcheries within each subbasin would be evaluated independently in a

similar way.

At the subbasin level, the evaluation would use a separate set of performance standards

and indicators to assess how well the composite of all hatchery activities within the

subbasin addresses subbasin-level objectives. All subbasins would be evaluated

independently in a similar way. To ensure internal consistency in the evaluation,

individual subbasin-level objectives must be consistent with province and basinwide

objectives.

At the subbasin, province, and basin levels the strategy for meeting each set of objectives

might include a variety of activities involving several of the four H’s (Habitat,

Hydropower, Harvest, and Hatcheries). The role that artificial production is to play in

meeting objectives needs to be articulated clearly at each level in the hierarchy.
ISAB 2000-2: Review of APR Performance Standards and Indicators

5

In summary, to develop a truly effective evaluation process for the artificial production

program in the Columbia River Basin, assessments at all hierarchical levels must be

integrated. To accomplish that task, a distinct set of clearly articulated objectives, as well

as a set of appropriate performance standards and indicators must be developed for each

level of the hierarchy. These should recognize the shift from small-scale tactical questions

for an individual hatchery to larger-scale strategic questions at the province and basin

level. Articulation of the objectives and performance standards and indicators at all four

hierarchical levels is paramount for any true evaluation to proceed.

To help guide future efforts to design the evaluation of artificial production in the basin,

we provide below a general review of the performance standards and indicators in the

APR. Those performance standards and indicators were designed only to address

hatchery-level performance, but they fall short of that limited goal. We also provide a

more in-depth critique for two of the performance standards and indicators in the APR.

Review of APR Performance Standards and Indicators:

In general we find that the description of the standards and indicators is confusing. It is

unclear how the current standards and indicators are to be used, in general or at the level

of an individual hatchery. Their relationship to the program framework’s APR policies is

also unclear. The APR needs to lead the reader through the logic of the proposed

evaluation process at the appropriate spatial scales. To help correct these problems, we

offer the following suggestions:

1. The breakdown of standards (and their associated indicators) into two categories,

benefits and risks, is unnecessary. We realize from a review of the facilitator’s report that

this breakdown was intended to reflect both the purpose (benefit) and the possible

detrimental effects (risks) of the artificial production activity. We recommend

streamlining this list of standards into a single body, but one developed for each level of

spatial hierarchy (hatchery, subbasin, province, and basin).

2. The language used to describe each standard is not typical; the end result is that each

of the "standards" is not truly interpretable as a standard. In addition, a number of cryptic

bullet points are often included without sufficiently indicating their relationship to the
ISAB 2000-2: Review of APR Performance Standards and Indicators

6

standard. A standard is a quantifiable state or condition described in such a way that it is

easy to determine whether or not it is being met. We provide specific examples below.

The same confusion exists for the performance indicators. Indicators are a list of

measurable metrics that bear directly on the quantitative determination as to whether or

not the standard is being met. Again, we provide specific examples below.

3. Construction of performance standards and indicators needs to take into account the

five purposes for artificial production: augmentation, mitigation, restoration,

preservation/conservation, and research.

4. As written, the standards and indicators appear to be the products of a facilitated

session. Synthesis of these ideas is necessary to convert them to reader-friendly prose

that states clearly each standard and the appropriate set of performance indicators that

would be used in the evaluation process.

5. As indicated previously, performance standards and indicators must be derived from

the objectives at each level in the spatial hierarchy.

We examine two standards and their indicators from the APR to illustrate the general

problems we found in the entire set. Because we believe that substantial revision of the

standards and indicators is likely to occur in the near future, we do not feel that it would

be particularly useful at this point to comment in depth on them all. We would be happy,

however, to participate or assist further in development of the hierarchical objectives and

performance standards and indicators.

Example 1:

Performance Standard #1 (under Benefits):

This currently is written as:

"Provide predictable, stable and increased harvest opportunity” (with a series of four

bullet points - that are unclear as to their purpose). There is no expression of any

standard to be met here, but rather only some vague opportunity. In addition, this

standard, as written, is inconsistent with APR Policy number three and the added
ISAB 2000-2: Review of APR Performance Standards and Indicators

7

explanation: “The performance of artificial production programs should mirror the

dynamics and behavior of the larger system. Expectations of constancy in either returns

or management are unrealistic.”

This standard could be revised as follows,

"Hatchery production of adult spring chinook meets the target production goal for

adults as established in the subbasin plan."

In this standard, an actual numerical value is clearly articulated.

Performance Indicators as given for the above:

This currently is written as: "Predictable, stable, and increased harvest opportunities met.

Managed for.... (with a long series of statements that are confusing as to how they

actually relate to potential indicators of performance - even knowing that all procedures

for setting target goals may not be in place). – Stated as is, this is close to a standard, not

an indicator designed to measure it.

This could be written as: "1. The goals in the subbasin plan for production of adults from

Hatchery A are ....... [ state clear, feasible, and acceptable numerical values for

production goals]. 2. Production methodology in Hatchery A adheres to genetic and

disease control protocols on file in the Office of ..., followed by a variety of other clearly

stated indicators that fit this specific standard. As they stand now in the current

document, indicators range from some that resemble standards to some that actually are a

listing of interesting variables to monitor, to some that more closely resemble a suggested

list of research topics for the future.

Example 2:

Performance Standard #3 (under Risks):

This currently is written as:

"Assess detrimental genetic impacts among hatchery vs. wild where interaction

exists".

Again, this is not a standard, but rather a recommended action.
ISAB 2000-2: Review of APR Performance Standards and Indicators

8

This could be written as:

"Hatchery production results in no interbreeding with wild stocks." (This example

represents the most conservative approach; the standard could be less restrictive.)

Performance Indicators as given for the above:

This currently is written as: "A. Initially, it is assumed that stray rate is a surrogate for a

thorough and more complex measurement of genetic impact. More specific...(followed

by a list of potential research/monitoring-type activities, some of which like "4.

Implemented an appropriate experimental design to quantitatively measure outbreeding

depression." are ambitious research programs for any hatchery to institute de novo, and

do not represent an "indicator" of performance).

This could be written as: “the composition of the spawning population has 0%

hatchery fish in it” (This example represents the most conservative approach; the

standard could be less restrictive.)

Summary

We were asked to address three specific questions relating to the draft performance

standards and indicators presented in the APR. In this report, we recommend an

approach to evaluating artificial production that integrates objectives and activities at four

hierarchical levels: hatchery, subbasin, province, and basin. We also suggest the need for

developing a set of performance standards and indicators to use at each level in the

hierarchy, depending upon which of the five purposes (augmentation, mitigation,

restoration, preservation/conservation, and research) are being addressed. Finally we

reviewed the set of performance standards and indicators included in the APR.

The ISAB was asked to answer the following specific questions:

Question 1. Are the draft performance standards and associated indicators the

appropriate tools to periodically evaluate the effects of individual artificial production

programs for the purpose of determining whether the principles, policies, and purposes in

the Artificial Production Report (APR) are being fulfilled?
ISAB 2000-2: Review of APR Performance Standards and Indicators

9

Answer: Performance standards and indicators are appropriate tools to use in the

evaluation of artificial production programs. The set of draft performance standards and

indicators, provided in the APR, however, need substantial revision before they would be

useful. It should be recognized that each of the five purposes for artificial production

(augmentation, mitigation, restoration, preservation/conservation, and research) would

require a different set of performance standards and indicators. It also needs to be

recognized that the draft performance standards and indicators included in the APR were

aimed primarily at the hatchery-level evaluations. Performance standards and indicators

are also needed for the subbasin, province and basinwide levels.

Question 2. Are the draft performance standards and associated indicators the

appropriate tools to adequately evaluate the effects of the artificial production activities in

the basin?

Answer: Again, a set of performance standards and indicators is an appropriate tool to

use in the evaluation of artificial production programs. It must be stressed, however, that

“to adequately evaluate the effects of the artificial production activities in the basin”, a set

of performance standards and indicators must be developed for each hierarchical level.

There could be as many as twenty generic sets of performance standards and indicators,

one for each of the five purposes for artificial production (augmentation, mitigation,

restoration, preservation/conservation, and research) at each of the four hierarchical levels

of organization (hatchery, subbasin, province, and basin). The generic performance

standards would then need to be made specific for the individual hatcheries, subbasins,

and provinces.

Question 3. If a performance standard, indicator or other means of measurement is not

the most appropriate tool for this purpose, what other standard or indicator would you

recommend?

Answer: To develop a truly effective evaluation process for the artificial production

program in the Columbia River Basin, assessments at multiple hierarchical levels must be

integrated. To accomplish that task, a set of clearly articulated objectives, as well as a set

of appropriate performance standards and indicators must be developed for each level of

that hierarchy (and taking into account the five different purposes for artificial

production). The evaluation of any hatchery requires the clear articulation of a set of
ISAB 2000-2: Review of APR Performance Standards and Indicators

10

production goals for use in meeting hatchery, subbasin, province, and basinwide

objectives. Initiating the development of multiple sets of performance standards and

indicators, however, should proceed systematically and concurrently with the articulation

of integrated programmatic objectives at all four hierarchical levels.

We hope this review has been helpful. We believe that development of these draft

performance standards and indicators was an important first step in a very important

process. As such, we would be happy to assist in its further development in whatever

ways we can.

________________________________________

w:\em\ww\isab\projects\2000-2 isab apr psi.doc