Independent Scientific Advisory Board

for the Northwest Power Planning Council

and the National Marine Fisheries Service

851 SW 6th Avenue, Suite 1100

Portland, Oregon 97204

Model Synthesis Report

An Analysis of Decision Support Tools Used in

Columbia River Basin Salmon Management

Executive Summary

March 2, 2001

ISAB 2001-1a

Peter A. Bisson

Charles C. Coutant

Daniel Goodman

Robert Gramling

Dennis Lettenmaier

James Lichatowich

Eric Loudenslager

William Liss

Lyman McDonald

David Philipp

Brian Riddell
ISAB 2001-1a Model Synthesis Report: Executive Summary

i

ISAB Model Synthesis Report

Contents of Executive Summary:

Purpose of the Project...................................................................................... 1

Synthesis of the Answers to the ISAB’s Questions........................................... 2

Conclusions...................................................................................................... 6

Recommendations............................................................................................ 9
ISAB 2001-1a Model Synthesis Report: Executive Summary

1

Executive Summary

Purpose of the Project

The major analytical efforts underway to support decision-making for salmon

restoration in the Columbia Basin include NMFS’ Cumulative Risk Initiative (CRI), the

Plan for Analysis and Testing Hypotheses (PATH), the Northwest Power Planning

Council’s use of the Ecosystem Diagnosis and Treatment protocol (EDT), the United

States Forest Service’s and Bureau of Land Management’s effort undertaken for the

Interior Columbia River Basin Ecosystem Management Plan Bayesian Belief Network

(ICBEMP-BBN), and the Columbia River Intertribal Fish Commission’s COHORT

model (CRITFC). Within the region, conflict has developed over the role these models

should play in the decision-making process and the need for multiple modeling

approaches. Decision-making could be hampered if the region becomes engulfed in a

“collision of models,” with various interest groups and agencies advocating the

conclusions that best support their interests and mandates. The ISAB undertook this

project to clarify the questions or problems that each model was designed to address, to

provide an overview of each of the models and a synthesis that describes both consensus

conclusions and areas of disagreement, and to assess the roles models play in the

decision-making process.

There are now at least three distinct philosophies of modeling represented in the

models available for decision support in Columbia Basin salmon management. These

philosophies are: (1) decision-analysis, embodied most clearly in PATH-FLUSH and

PATH-CRiSP, (2) statistical, embodied most clearly in CRI, and (3) expert-system,

embodied most clearly in ICBEMP-BBN and EDT.

Of the three approaches, the decision-analysis approach is most closely directed at

providing management advice, and it is the most formal about factoring uncertainty into

the analysis. This modeling approach has the potential to be the most useful to decisionmakers.

The decision-analysis approach, however, is very difficult to implement

successfully. Its success depends crucially on the engagement of the actual decisionmakers in framing the questions that need to be answered, identifying the management

options that are under consideration, and in defining the "values" put on the various

possible outcomes. Such engagement and communication is difficult to achieve in the

institutional setting of Columbia Basin salmon management, where there is so much

fragmentation of decision-making authority.

The statistical approach is scientifically the most classical of the three, and can

operate with a large degree of detachment from policy. It proceeds by testing hypotheses

and estimating life history parameters with available data. This has the advantages of

clarity, rigor, and empirical objectivity. The limitation of this approach is that the scope

of the questions that can be answered with satisfactory conclusiveness is restricted by the
ISAB 2001-1a Model Synthesis Report: Executive Summary

2

availability of data. In a domain that is data-poor, too many pressing questions may go

unanswered.

Expert-system approaches fill gaps in the data with expert opinion. In the context

of the salmon problem, expert opinion allows consideration of the most concrete menu of

specific options for actual management. Expert opinion is, admittedly, a weaker basis for

scientific prediction than is a mathematical relationship validated with an archive of

quantitative empirical data. It is important to recognize, however, that at the level of

spatial resolution and environmental detail required to make salmon management

decisions, there are no available validated mathematical formulae for predicting reliably

the effects of management actions on salmon, and there is no adequate data archive for

deriving such formulae. In this light, the expert-system approach may well be a

reasonable and practical method for providing tentative answers to some management

questions that need to be addressed quickly.

Synthesis of the Answers to the ISAB’s Questions

In undertaking this project, the ISAB requested the assistance of the principal

scientists involved in each analytical approach. We asked that they 1) provide written

responses to a series of questions about their modeling approach, 2) give an oral

presentation of their responses and participate in a discussion of the responses with the

ISAB, and 3) participate in a panel discussion with the ISAB and the representatives of

each of the modeling approaches. The following is a synthesis of their responses to our

list of questions. A more detailed summary is presented later in the report (Table 1) and

the actual written responses are included in Appendix III.

What is the purpose of your modeling effort? What questions or problems were your

models designed to address?

Purposes of the decision support systems (models) are different. Two (PATHFLUSH and PATH-CRiSP) are primarily designed only to evaluate Snake River salmon

recovery actions including breaching the lower four Snake River dams, increased

transportation, and reduced harvest. Two (ICBEMP-BBN and EDT) are primarily

designed to evaluate large-scale habitat management alternatives across the landscape of

the Columbia River Basin and three (CRITFC, CRI and PATH-EM) are primarily for

prediction of trends in populations of salmonids based on counts of fish and rates of

change in population parameters.

What kinds of information or data are needed to run your model?

Data needs are different, because the purposes of the models are different. PATHFLUSH and PATH-CRiSP models concentrate on estimating survival based on flow and

other data as fish enter the lower Snake River. ICBEMP-BBN and EDT concentrate on

predicted effects of habitat changes over large areas based on spatial habitat data.

CRITFC, CRI, and PATH-EM concentrate on predicted trends in sizes of populations

based on fish counts and rates of change of population parameters (e.g., survival rates) in
ISAB 2001-1a Model Synthesis Report: Executive Summary

3

the presence of various management actions. One characteristic common to all of the

models is that information required to run the models includes not only primary data

measured by documented procedures, but also derived values each depending on yet

other (perhaps undocumented) assumptions and ‘models.’

What are the assumptions of your model?

Not only are the purposes and data needs different among the models; the models

depend on different assumptions made by the respective developers concerning the state

of nature. Some of the key assumptions made are highlighted in Table 1, and the

presenters described others (Appendix III). Two of the models, ICBEMP-BBN and EDT,

are “expert systems” depending heavily on opinions (i.e., assumptions) expressed by

panels of experts. All of the models implicitly assume that future climate and ocean

conditions will be like those in the recent past.

What are the strengths and weaknesses of your model?

Given that purposes, data needs, and assumptions are different, it is not surprising

that the models have different strengths and weaknesses (Table 1). Some involve

complex interactions of physical and biological factors (ICBEMP-BBN, EDT, PATHFLUSH, and PATH-CRiSP), others (CRI and CRITFC) depend on relatively simple

mathematical procedures that can be easily programmed and duplicated, while PATHEM fits somewhere in-between. Simplicity and complexity are each simultaneously a

strength and a weakness. The simpler models allow relatively more direct point estimates

of life history parameters under a given set of assumptions, but lack the ability of the

complex models to help understand mechanisms of cause and effect relationships. The

more complex models deal better with the comparison of management alternatives, but

usually can only rank alternatives on their expected benefits. It would be nice of course

to have the advantages of all the decision support systems in one ‘model’, but this seems

to be impossible.

How does your model address uncertainty?

Only one model (CRI) reported an attempt to deal with variation in the data due to

sampling (observational error) so that confidence intervals can be placed on predicted

values. However, CRI was not totally successful in this attempt and along with EDT,

PATH-EM, and CRITFC, performed sensitivity analysis (e.g. presented "best case-worst

case" outcomes of actions). The other models, ICBEMP-BBN, PATH-FLUSH, and

PATH-CRiSP, utilized probabilistic arguments either by standard procedures or Bayesian

methods to assign probabilities to expected outcomes. None of the models address the

issue of the feasibility of implementing management actions in the face of social and

economic constraints.
ISAB 2001-1a Model Synthesis Report: Executive Summary

4

All models make predictions. Why do you think your model’s predictions are accurate?

In comparison of major management alternatives, all of the models depend

primarily on the relative rank of predictions, not accuracy of the actual predicted

numerical values. The rank orders of effects of alternative actions were judged to be

accurate by the model presenters.

How does your modeling effort relate to or contrast with the other modeling efforts?

The modeling efforts have very different objectives. Both EDT and ICBEMPBBN rely on expert opinion to rank effects of habitat changes over large areas, but EDT

gives finer scale analyses. CRI, PATH-EM, and CRITFC attempt to predict the trends of

stocks of fish, but with different approaches and information. PATH-FLUSH and PATHCRiSP concentrate on analysis of only two Snake River ESUs. Never-the-less, the

general conclusions when faced with the same objectives and when based on more-orless the same information are often in close agreement with respect to the predicted rank

order of effects of management alternatives. As PATH-FLUSH concluded “The decision

analysis of PATH showed which actions were most likely to benefit the stocks under the

widest range of assumptions. Analyses using the same data and the Leslie matrix model

advocated by CRI give strikingly similar results about the relative effectiveness of the

proposed alternatives in preventing extinction and achieving recovery of listed Snake

River stocks.”

What is the efficacy of dam breaching or draw down to natural river levels for delisting

ESA species and restoring diverse and productive populations of native fishes throughout

the Columbia River Basin?

There is general agreement among the models that breaching dams is effective in

restoring mainstem habitats to the benefit of listed fall chinook and breaching would

provide significant benefits to both Snake River fall and spring listed stocks. All models

also agree with the obvious and important conclusion that breaching dams on the Lower

Snake River will do little or nothing to help the other ten listed ESA species. With respect

to the Snake River spring chinook population, there is general agreement among the

PATH models that dam breaching or draw down are better than transportation (with the

dams in place) as a recovery measure. CRI and CRITFC conclude that survival gains

from other factors such as tributary and estuarine habitat improvement are needed to

allow the Snake River spring chinook population to rebuild, even with dam breaching.

ICBEMP-BBN is not useful in judging the efficacy of dam breaching.

What is the efficacy of hatcheries for delisting ESA species and restoring diverse and

productive populations of native fishes throughout the Columbia River Basin?

No satisfactory conclusions were given concerning the efficacy of hatcheries for

delisting ESA species. Only EDT and PATH-EM seems to have taken an initial look at

modeling the effect of reduced hatchery production (for wild steelhead in the Snake

River).
ISAB 2001-1a Model Synthesis Report: Executive Summary

5

What is the allocation of harvest and harvest levels needed to delist ESA species and

restore diverse and productive populations of native fishes throughout the Columbia

River Basin?

The effect of changing harvest received varied and limited attention by the

modeling efforts. CRI concluded that reducing harvest would allow only three ESUs (Fall

Chinook in the Snake River, Upper Willamette Chinook, Lower Columbia Chinook) to

grow, a conclusion not much different from the PATH determination that harvests were

not significant contributors to declines in survival of Snake River Basin populations after

1974, or the CRITFC note that elimination of all harvest, by itself, is not sufficient to

maintain or rebuild spring chinook populations in the Snake River or above Priest Rapids

dam. ICBEMP-BBN, EDT, and PATH-EM have not attempted to assess impacts of

harvest alternatives on ESA stocks.

What is the efficacy of restoration of tributary and mainstem habitat for delisting ESA

species and restoring diverse and productive populations of native fishes throughout the

Columbia River Basin?

The two broad scale habitat models, ICBEMP-BBN and EDT, both agree that

over the long term improving tributary habitat is effective in maintaining or restoring

abundance and distribution of tributary populations. The rather obvious conclusion is

supported that breaching dams was effective in restoring mainstem habitats to the benefit

of listed Snake River fall chinook. CRI related salmon productivity to characteristics of

tributary habitat. PATH, in contrast, concluded that tributary habitat degradation was not

responsible for the decline in Snake River stocks following the construction of the lower

Snake River dams.

What advice would you give decision-makers on how they should use your model to

support decisions regarding salmon recovery in the Columbia River Basin?

The decision support systems have different strengths and weaknesses and policymakers would be well served by drawing on all of the available analytical tools. Six of

the seven presenters implied that their model should be used in conjunction with other

models, because in the words of one “No model does it all for most decisions. Use EDT

in conjunction with other models such as the CRI, PATH, and CRITFC tools. Take

advantage of the strengths of each model system and appreciate the different perspectives

afforded by each.” The ISAB wholeheartedly endorses this recommendation. Results

from a model, given knowledge of its assumptions and quality of data used, cannot

decrease the information available to a decision-maker.
ISAB 2001-1a Model Synthesis Report: Executive Summary

6

Conclusions

Conclusion 1: None of the models presently in use in the Columbia Basin is complete

enough to serve as the sole decision support tool for the region.

The models available to support decisions in the Columbia River Basin serve

different functions and all have strengths and weaknesses. The models differ in the

problems they are attempting to address, the analytical approaches to the problems, the

assumptions underlying each of the approaches, the quantity and quality of the available

data, and the rigor with which they deal with the complex life cycles and habitats of the

species.

PATH scores high for its attempt to operate in a decision context that was explicit

about uncertainties, and for attempting to formally quantify uncertainty in a reasonably

sophisticated way. Where PATH appeared weak was in the very narrow range of factors

that it took into consideration, and in the tortuous model structure that posed an obstacle

to diagnosis of model behavior. Notwithstanding a large amount of good work, PATH

was vulnerable to the criticism that it had not answered the question asked of it, and that

it was trying to answer policy-sensitive questions that had not been asked of it. It was not

entirely clear, however, who was in charge of PATH, or to whom PATH was supposed to

answer. Furthermore, because of its "consensus" structure, PATH was notoriously slow

in operation, and it generated unwieldy volumes of reports.

CRI scores high for clarity and tractability of model structure, but it lacks

statistical treatment to quantify uncertainty. Although CRI encompasses a broader range

of factors than the original PATH models, it still appears deficient in necessary detail for

habitat variables and spatial and temporal structure. Both CRI and PATH rely on derived

quantities in place of actual data, in ways that confound a proper uncertainty analysis.

CRI clearly is investing heavily in attempts to remedy these deficiencies.

EDT scores very high marks for comprehensiveness of factors taken into account,

and for spatial and temporal detail in the data it uses. It is not yet clear whether EDT is

sufficiently rigorous about maintaining the separation between actual measurements and

derived quantities, or in quantifying the uncertainty of the derived quantities. The

predictive modeling component of EDT is essentially an expert system. The process of

eliciting expert judgment may include obtaining opinion about certainty, but that too is

expert judgment, rather than a statistical comparison to actual measurements. So it looks

as if EDT does not yet include a satisfactory treatment of uncertainty.

ICBEMP-BBN like EDT gets high marks for comprehensiveness and for spatial

and temporal detail. ICBEMP-BBN went one important step beyond EDT by formally

eliciting expert judgment about uncertainty and propagating this uncertainty through the

model predictions. ICBEMP-BBN was constructed as a Bayesian Belief Network, which

confers the advantage of explicit representation of uncertainty as probabilities. The

Bayesian statistical framework lends itself to value-of-information calculations that can

help guide decisions about future monitoring, ground truthing, or experimental
ISAB 2001-1a Model Synthesis Report: Executive Summary

7

interventions to help resolve critical uncertainties. ICBEMP-BBN did not avail itself of

this potential, but it is an attractive option that should be considered as EDT evolves

toward calibration and validation.

Both ICBEMP-BBN and CRI seemed to enjoy very successful communication

with their respective decision-makers. In both cases, scientists and decision-makers were

part of the same organization. The decision questions that the modeling efforts were

intended to inform were well-defined in advance of the modeling project, the models

were directed at those questions, and the decision-makers seemed not to experience

difficulty in understanding and using the model results.

Among the Columbia River basin salmon modeling efforts, the PATH

experimental management (PATH-EM) report represents the most substantive attempt, to

date, to provide an analysis of the prospects for actual Adaptive Management. The

analysis described specific experiments and provided a preliminary assessment of the

types of information that the experiments would provide, and how much time they would

require for completion. There seems to be a temptation, in many quarters, to co-opt the

term “adaptive management” merely to put a positive spin on vague management plans.

Thus, it is important that this kind of technical analysis of adaptive management resume

as an ongoing enterprise connected to actual decision making in the Basin.

Conclusion 2: The models are best at ranking the expected effects of management

alternatives.

The general conclusions of the models are often in close agreement with respect

to the predicted rank of management alternatives when addressing similar problems and

using the same data sets. The models, however, are not good at giving absolute numerical

predictions and they do a poor job of accurately estimating what the policy-makers may

need most, namely, a credible scientific analysis of the probability (feasibility) that some

measurable degree of salmon recovery will be achieved with any particular management

action.

Scientists can help environmental decision processes to cope better with

uncertainty by explicitly quantifying the relevant uncertainty. Statements of the

respective probabilities of alternative scenarios are a natural way to communicate

uncertainty when the decision is essentially placing a bet about which scenario actually

will materialize.

Conclusion 3: All the modeling efforts are severely constrained by lack of data.

Modeling controversies in the salmon arena have largely been an unproductive

distraction from the real scientific problem of inadequacy of the available data for

addressing many of the important management questions. Some of the debate that now

centers on competing models could be resolved with the right data. The present paucity

of data creates more scope for alternative assumptions in the models. Sophisticated,

responsible modeling takes all the plausible alternative assumptions into account with
ISAB 2001-1a Model Synthesis Report: Executive Summary

8

weighting according to their respective concordance with the data that are available. This

need not lead to "modeling wars."

Where scientific leadership and institutional innovation needs to be exercised is

in the prioritization, design, and implementation of large-scale monitoring linked to

management experiments. There is at present no institutional center of authority for

addressing the prioritization; design and coordination issues for large-scale monitoring

linked to management experiments.

Conclusion 4: Decision-makers would be well served by drawing on all the available

analytical tools.

Decision-makers would benefit by focusing on areas of consensus among the

models or the weight of evidence provided collectively by the models. Areas of

disagreement among the models may pinpoint uncertainties that require further

investigation. In considering how results of models make their way into the decisionmaking process, it is helpful to recall the roles of models. They provide ways of

organizing and communicating information, generating hypotheses, and pin pointing the

crucial gaps in information. The modeling efforts are not ends in themselves; they are not

final, definitive answers, but rather they are ongoing processes for continuously

increasing knowledge.

Conclusion 5: Effective communication between decision-makers and scientists is

essential if scientific results are to play an integral role in the decisionmaking process.

Decision-makers need to understand what science is able to accomplish relative to

a particular problem, given data limitations, inherent environmental variability,

uncertainty, and so forth. Decision-makers need to frame questions that are appropriate

for science to address and to make their needs clear to scientists. To maintain the trust

and support of scientists and the credibility of the decision-making process, decisionmakers should clearly explain the basis of a decision to scientists and the public.

To prevent scientific debates over the models from encumbering the decision

process, it is crucial for decision-makers to understand both the capabilities and

limitations of models. Without an understanding of model capabilities and limitations

there is the danger that decision-makers will develop unrealistic expectations of the

models. Models constitute a way of organizing and communicating information. They

provide a systematic way of predicting outcomes of management interventions,

identifying what cannot be reliably predicted and quantifying uncertainty. Models can

generate useful hypotheses and identify crucial gaps in knowledge. These functions

provide a valuable guide for setting priorities for data collection and suggesting new

experiments to resolve critical uncertainties.
ISAB 2001-1a Model Synthesis Report: Executive Summary

9

Scientists need to comprehend the needs of the decision-maker and understand

that, although science plays a critical role in formulating decisions, decision-makers also

must weigh the social, cultural, and legal implications of a pending decision. Scientists

need to inform decision-makers of the full suite of management alternatives that address

a particular question or problem and the uncertainty associated with each alternative. The

nature and extent of the uncertainties have to be explained in language that is meaningful

to the decision-maker. This is especially important when the uncertainty is large, as is

common in salmon management problems. If decision-makers are not adequately

informed, their expectations are not met, or they are faced with uncertainty that is a

consequence of strong disagreement among scientists over appropriate courses of action

for recovery, the decision-making process could be driven largely by social, legal, and

economic factors, to the possible exclusion of much of the available science, or a decision

will be deferred pending further research.

Recommendations

Recommendation 1: At present, the region should fund specific groups to undertake

specific modeling projects, not large collaborative modeling efforts.

Achieving consensus among scientists with widely divergent views has proven to

be difficult in large, collaborative modeling projects. These large modeling efforts may

not be necessary because the kinds of models that are actually justifiable are not really

that complicated, given the amount of data that are available. Small groups of researchers

should be adequate to pursue development of the models. And the coherence of vision of

a small group may encourage clarity in the resulting model. That coherence is very

difficult to achieve as the group gets large. The correct way to proceed with modeling

needs is to fund specific groups to undertake specific modeling projects.

There would be merit to funding several groups to pursue modeling questions.

There would even be merit to a degree of overlap, so that the same data sets might be

analyzed from somewhat different technical perspectives, which would encourage the

evolution of scientific insights. The actual dollar cost of modeling, as a fraction of the

overall salmon recovery budget, has been very small, so this is not the place to look for

cost cutting.

Overall, the region needs modeling efforts that collectively combine the best

features of all the models we have reviewed. Features of ICBEMP-BBN and EDT could

be used to formulate working hypotheses concerning, for example, causes of salmonid

declines and potential effectiveness of management interventions. Features of ICBEMPBBN could be used to communicate the uncertainty of the working hypotheses. Features

of CRI could be used to test hypotheses wherever data were available. Features of PATH

could be used to quantify the uncertainty of the tests of the hypotheses and to place

recommendations for management in a risk assessment context. Furthermore, CRI might

serve as an inspiration for clarity, rigor, and openness.
ISAB 2001-1a Model Synthesis Report: Executive Summary

10

Recommendation 2: The region should develop and implement a region-wide monitoring

and evaluation program.

The belief that Columbia River salmon management constitutes an expensive

failure of “salmon science” is a vastly oversimplified perception. Most of the money

spent on salmon recovery has been spent on management actions, not on science. The

science that was conducted has contributed substantial knowledge that is relevant to

salmon recovery, but this knowledge has not always found its way into the decisionmaking process. One of the critical shortcomings that hamper salmon science is the

shortage of high quality data. There are many important questions in salmon management

that are going unanswered for lack of data, and some of the debate that now centers on

competing models could be resolved with the right data. Where scientific leadership and

institutional innovation needs to be exercised is in the prioritization, design, and

implementation of large-scale monitoring linked to management experiments.

Recommendation 3: Scientific results should be published in the peer-reviewed literature.

One pathway toward resolution of scientific disagreement is through publication

in the peer-reviewed scientific literature, and provision of access to model code and to

data files allowing independent verification. Data access should include access to the

original primary data and metadata. If there is not access to the primary data, derived

quantities, when treated as if they were data, may carry error that escapes scrutiny.

Publication in the scientific literature establishes credibility for the use of science in

Columbia Basin decision-making. A greater reliance on the mechanisms of normal

scientific discourse might also reduce some of the contentiousness that has characterized

the history of scientific debate over key issues in Columbia Basin salmon management.

Recommendation 4: The region should develop effective ways to improve communication

between scientists and decision-makers.

To facilitate communication between scientists and managers a formal

institutional mechanism for synthesizing scientific results and clarifying the interpretation

of the various salmon models for the policy makers should be developed.

If a modeling effort is motivated by a desire to contribute to a particular decision, it is

especially helpful to initially invest in enough communication to ensure that the model

really is addressing the right question. Scientists can work with decision-makers in

crafting decision rules that get formalized before the analysis is undertaken. Decision

rules define what measurements will be made, what statistical operations will be

performed on the data, and what threshold magnitudes of estimated quantities at specified

levels of certainty will serve as criteria for the decision. Such specifications help remove

ambiguity from the way science is used in the decision process. Committing to these

specifications in advance helps dispel suspicions that the analysis may be manipulated to

achieve a particular outcome. If institutional trust does not embrace a particular model,

the need to establish credibility can stand in the way of getting a model's results

considered in the course of a decision.
ISAB 2001-1a Model Synthesis Report: Executive Summary

11

________________________________________

w:\em\ww\isab\1 isab final reports\isab 2001-1a model synth exec summary.doc