May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

1

Summary of Results from GCDAMP TWG 

Multi-Attribute Evaluation Workshop 

Phoenix (May 28, 2003) 

 Lee Failing 

Josh Korman 

Carl Walters 

1.0 Background and Purpose 

This project was initiated as a result of a desire by the Glen Canyon Dam Adaptive Management 

Program (GCDAMP) to develop quantitative targets for key resources against which the success 

of management actions and experiments could be evaluated. After preliminary scoping, the 

project team1

 concluded that the existence of trade-offs among key resource endpoints meant 

that a single-attribute approach to target setting would be inappropriate. The project team 

proposed that GCDAMP develop instead a multi-attribute trade-off analysis (MATA) that would 

serve as a framework for evaluating management options. In February 2003, the Technical 

Working Group (TWG) of the GCDAMP agreed to participate in a workshop to develop and test a 

multi-attribute evaluation framework. The objectives of the workshop were: 

– Gain an understanding of interactions among key resource endpoints 

– Expose key trade-offs and uncertainties 

– Gain experience with some structured methods for exposing values and preferences 

– Identify refinements to endpoints, attributes, options and modeling/estimation methods 

– Identify preliminary areas of agreement and disagreement 

In preparation for the workshop, the project team developed a draft evaluation framework, 

consisting of a small set of endpoints (resource outcomes of concern) and attributes

(quantitative metrics for assessing the impact of the management options on the endpoints). A 

preliminary set of management options was defined. A consequence table summarizing the 

expected impact of the options on the endpoints was prepared using existing information, 

models and expert judgment. 

2.0 Endpoints and Attributes 

The draft endpoints and attributes were refined with input from participating TWG members 

(Table 1). Additions or changes included: 

- combining sand deposition for camping beaches and archaeological sites into a single 

attribute (previously separated) 

- addition of long term beach sustainability as an attribute 

- removal of attribute for rainbow trout below Lees Ferry 

- addition of boating safety and accessibility (as affected by water level fluctuations) 

- modification of the calculation of the power/financial attribute so that power impacts 

include both revenue implications and the cost of purchasing capacity. 

Due to time limitations, the latter two changes were not incorporated into the pilot, but are 

noted for future analysis. 

 

1 The project team is headed by Ecometric Research from Vancouver, British Columbia. Team members 

include Josh Korman of Ecometric, Dr. Carl Walters of University of British Columbia, and Lee Failing of 

Compass Resource Management.
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

2

Table 1 Endpoints and Attributes

Endpoint Attribute Description 

Abundance 

(# fish relative to 

2003) 

Reports the incremental effect of management options on the expected 

abundance of humpback chub in the Little Colorado River. Number 

reported is a loss or gain relative to 2003 abundance. This attribute may 

be viewed as a surrogate for other native fish species. 

NATIVE FISH – 

Humpback 

Chub: LCR 

Probability of 

Extinction 

(%) 

In some cases, a management option may have a higher expected value 

for chub abundance, but also a higher risk of a negative outcome. This 

attribute reports the probability of extinction of humpback chub in the 

Little Colorado River by 2030. This attribute may be viewed as a 

surrogate for other native fish species. 

NATIVE FISH - 

HBC: 

Mainstem 

Probability of 

Establishing (%) 

Reports the probability of establishing a viable population of humpback 

chub in the mainstem Colorado River. 

Abundance 

(thousands) 

Reports expected abundance of fish > 150 mm in reach above Lees 

Ferry. 

Size (% of 2003) Reports expected size relative to 2003 above Lees Ferry 

SPORT FISH – 

RBT 

Probability of 

Major Decline (%) 

Reports the probability that the abundance of rainbow trout above Lees 

Ferry will decline by 50% or more. 

Beaches and Arch 

Sites 

(% of 1984 area)

Reports the expected area of sand bars in Marble Canyon. Increased 

sand deposition will improve the availability and quality of camping 

beaches and increase protection of archaeological sites from erosion. 

SAND 

DEPOSITION 

Sustainability of 

Sand Sources (%) 

Reports the probability that the estimated sand deposition can be 

maintained indefinitely. 

RIPARIAN / 

WILDLIFE 

Peregrine Falcon 

Abundance 

(% of 2003) 

Reports the expected incremental effect on peregrine falcon abundance 

as a result of management options. Can be used as a surrogate for other 

avifauna that depend on the cold clear aquatic ecosystem above Lees 

Ferry. 

BOATING 

ACCESS / 

SAFETY 

Magnitude of 

fluctuation 

Reports the magnitude of the daily or weekly water level fluctuation. 

Indicates the relative degree of safety and/or convenience for boaters. 

Annual Cost 

(million $ / year) 

Reports the average annual cost of a management option. For dam 

operation options, this involves an estimate of annual power costs 

relative to unconstrained operation; for non-power options, it involves 

estimating the levelized annual cost associated with construction and 

operation of non-power works. 

POWER / 

FINANCIAL 

SPA Issue? 

(Yes/No) 

Reports whether or not a management options creates issues with 

respect to the Colorado River Storage Project Act. It was argued that 

issues associated with other Acts (e.g. Grand Canyon Wilderness 

Protection Act) should also be included, or that this attributed be 

dropped. 
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

3

3.0 Management Options and Consequences 

Management options that were evaluated included: 

POWER — Flows optimized for power production, without regard to impacts of diurnal 

fluctuation 

MLFF — Continuation of current water operations and restrictions on flow variation 

TCD — Construction of temperature control device to warm the river, in conjunction with MLLF 

flow option 

SED PIPE — Augmentation of sediment inputs at Lee’s Ferry 

FLOW OPTION A — Steady flows in fall, with a January spike flow intended to conserve 

sediment 

FLOW OPTION B — Like OPTION A, but longer steady flow period (summer and fall) to benefit 

native fishes as well as conserve sediment 

FLOW OPTION C — Steady summer flow, fluctuating flow for rest of year (including fall), spike 

flow in January 

FLOW OPTION D — Like OPTION C, but bigger spike flow in January to deposit sand on eroding 

cultural sites 

TWG members reviewed and provided input on the preliminary management options. The draft 

consequence table was updated with input from TWG members (Table 2). It summarizes the 

estimated impacts of each option on each attribute. 
May 2003 

GCDAMP MATA Pilot: May 2003 Workshop Report

Table 2 Estimated Impacts of the Management Options on the Attributes
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

5

4.0 Ranking and Weighting Exercises 

After a review of the consequence table, stakeholders participated in a structured values 

elicitation process. There are three main reasons to use structured methods to elicit 

stakeholder values and preferences: 

- to increase the accuracy and consistency of individual stakeholder judgments 

- to increase the accountability and transparency of decisions by making the trade-offs made 

by stakeholders explicit 

- to provide focus for constructive deliberations and refinement of the options. 

There are many ways to elicit values. Different methods usually produce different results; no 

method is necessarily right. The use of multiple methods provides insight to the decision by 

thinking about it in different ways. By examining choices from different perspectives, 

stakeholders will have more confidence that their choices reflect their values, and are not the 

result of methodological bias. 

The methods used in the workshop were selected because: 

- They have a strong theoretical basis and are technically defensible; 

- They are simple to understand and easy to process, with a quick turnaround time; 

- They produce results in a format that support constructive deliberations; 

- They have a strong track record of success, having been used in support of stakeholder 

deliberations at nearly 20 hydroelectric facilities in British Columbia, Canada. 

For the GCDAMP pilot, stakeholders conducted two ranking and weighting exercises: 

- Direct Ranking 

- Swing Weighting 

In Direct Ranking, stakeholders were asked to rank and then score each management option 

directly. In swing weighting, they were asked to rank and weight each attribute. The term 

“swing” weighting is used because decision makers are asked to say which attribute they would 

most want to “swing-up” from its worst to its best value. This is important because in some 

cases an attribute may be important in a general sense, but the actual change in the attribute 

value that results from the choice among management options may be relatively insignificant; 

this should affect the weight assigned to it. 

Attribute weights were entered into the following equation that computed an overall score for 

each option: 

SCORE(a) = W1(x1a) + W2(x2a) + ...... 

Where: 

SCORE(a) = the calculated score for a management option (e.g. ‘a’) 

W1, W2... = the weight of an attribute 

x1, x2... = the scaled impact of a given option on each attribute 

Ranks for each management option for each stakeholder were then derived. 
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

6

5.0 Results 

This exercise was a pilot, and as a result, caution should be used in drawing any conclusions 

about selecting or rejecting management options. However, the following outlines the kinds of 

insights the analysis can provide to decision makers. 

Figure 1 provides an example from a single stakeholder comparing their ranks assigned by the 

direct method and ranks assigned by the swing weighting method. Options ranked the same by 

both methods fall on or near the 45 degree line. Options that fall far from the 45 degree line 

should trigger a re-examination of that alternative by the stakeholder. For example, from 

Figure 1 we see that Stakeholder 2’s ranks are quite consistent across the two methods except 

for Flow Option B. This option is ranked very low by the direct method, but is ranked number 

one by the weighted method. While this does not necessarily mean that the direct rank is 

wrong, it may indicate any of a number of problems, such as: 

- mixing up the options in the direct ranking (common when there are many options); 

- overlooking some elements of performance in the direct ranking (common when there are 

many attributes) 

- overlooking options that are less controversial or less visible (reflecting a tendency to 

spend more discussion time on options with either vocal champions or vocal opponents). 

Alternatively the direct ranking may be a more accurate reflection of the stakeholder’s values 

if the attributes do not adequately capture all the important elements of performance. The 

intent of the multi-method approach is therefore not to say that one method is better than 

another, but to expose inconsistencies, clarify the rationale for choices, and improve the 

transparency and accountability of decisions. 

Figure 1 Comparison of Ranks by Direct and by Swing Weighting Methods for Stakeholder 2 

Comparison of Direct Ranking versus Ranking based on Swing 

Weights 

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Rank by Swing Weights

Rank by Direct

Alternatives

MLFF

TCD w MLFF

Sed Pipe - Sand

Flow A

Flow B

Flow C

Flow D

POWER - No Action

Sed Pipe - Turbidity

45 degree line

Alternatives located on the 45 

degree line have the same rank 

using either method of 

evaluation. 

Alternatives located off the 45 

degree line have different ranks 

depending on the evaluation 

method used.
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

7

Figure 2 shows the range and distribution of weights assigned to each attribute by the group of 

stakeholders. Some clear groupings of weights can be seen in this figure, as well as significant 

outliers. All participants placed a similar and relatively high weight on humpback chub in the 

LCR, but one participant placed a much lower value on establishing a mainstem population. 

This highlights an area where further dialogue may be constructive. The low weight may reflect 

the participant’s true values; however, it also may be a case where further dialogue among 

participants would increase understanding of the relative importance of a second population, 

and values may converge. In other cases, there is general agreement about relative values (for 

example, wildlife and boating access/safety are consistently weighted lower than chub, sand 

and Lees Ferry rainbow trout). Note first that this reflects the weight assigned given the swing 

across the range of possible outcomes, which according to the impact estimates do not include 

catastrophic outcomes for either of these attributes. This general agreement suggests that 

further debate about the relative value of these resources may not be necessary. 

Figure 2 Range and Distribution of Weights Across Stakeholders 

0

5

10

15

20

25

30

35

02468

HBC LCR

HBC Mainstem

Rainbow

Sand

Wildlife

Power

Boating

Figure 3 summarizes the ranks assigned by stakeholders to each option by each method. 

Options ranked 1 or 2 are colored green, 3 or 4 are yellow, 5, 6, or 7 are white and 8 or 9 are 

red. From this we could conclude that the TCD option received a lot of support, both by direct 

and weighted methods and may be a candidate for further exploration. Direct ranks for Flow 

Option B were variable, but this option was ranked either first or second by all stakeholders by 

the weighted method, suggesting it is also a candidate for further consideration. Flow Option A 

while receiving few first or second place ranks, did not elicit a high degree of opposition. The 

remaining options did not score well by either method, and, in a real rather than pilot 

evaluation, might be eliminated from further consideration. 
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

8

Figure 3 Ranks Assigned to Alternatives by All Stakeholders by Direct and Swing Weighting 

Methods 

6.0 Conclusions and Next Steps 

This MATA exercise was prepared and executed as a pilot exercise to explore the multiattribute approach, test specific methods, and determine their usefulness for GCDAMP in 

future decision making. Limitations included: 

- limited stakeholder participation, with the result that the full range of values of the TWG 

membership was unlikely captured; 

- relatively short timeframe may not have allowed full discussion and understanding of the 

attributes and the significance of the impacts; 

- set of attributes may be incomplete; 

- set of management options was incomplete; 

- some important aspects of uncertainty associated with some options were not fully 

represented or discussed. 

The TWG should therefore use caution in drawing any conclusions about selecting or rejecting 

options on the basis of this analysis. However, as a test of methods, the pilot was quite useful. 

Participants expressed unanimous support for the MATA approach. Key benefits included: 

- making values of all stakeholders explicit; 

Stakeholders Method MLFF TCD w 

MLFF

Sed Pipe -

Sand Flow A Flow B Flow C Flow D POWER - No Action

Sed Pipe -

Turbidity

Direct 8 1 3 5 5 5 9 3 1

Swing 935217468

Direct 9 4 1 3 6 6 2 5 8

Swing 9 4 2 5 1 8 3 6 7

Direct 6 2 9 6 6 5 4 2 1

Swing 7 2 9 4 1 6 8 3 5

Direct 6 9 6 1 2 2 2 2 6

Swing 7 4 8 2 1 6 5 3 9

Direct 9 1 2 6 5 7 7 3 3

Swing 9 3 6 2 1 5 4 8 7

Direct 2 2 1 4 5 6 9 6 6

Swing 8 2 7 3 1 6 4 5 9

Direct 2 1 8 5 4 6 7 2 8

Swing 7 3 8 2 1 6 4 5 9

Direct 3 2 9 6 6 6 5 1 3

Swing 5 2 9 4 1 6 8 3 7

Direct 7 6 4 3 2 5 1 9 8

Swing 9 2 5 3 1 7 4 6 8

Direct 7 1 8 2 2 5 5 2 8

Swing 6 2 8 3 1 7 5 4 9

Direct 4 1 3 2 4 7 8 8 6

Swing 7 1 9 4 2 6 8 3 5

Direct 3 2 8 4 5 5 5 1 8

Swing 8 4 7 2 3 9 5 1 6

Direct 9 3 8 7 2 3 6 5 1

Swing 9 1 7 3 2 8 5 4 6

Direct 2 1 7 7 6 5 3 7 4

Swing 7 3 8 2 1 6 5 4 9

Direct 6 1 2 5 3 4 7 8 8

Swing 8 2 5 3 1 6 4 7 9 15

13

14

11

12

9

10

7

8

4

5

6

1

2

3
 May 2003 

GCDAMP MATA Pilot: 

May 2003 Workshop Report

9

- facilitating constructive dialogue about value differences; 

- distinguishing between disagreements caused by differences about facts and those caused 

by differences about values; 

- exposing uncertainty and the role of risk tolerance in making decisions and trade-offs; 

- focusing on the evaluation of management options, which some TWG members identified as 

the next step in the strategic planning process; 

- facilitating constructive dialogue between stakeholders and scientists, with emphasis on 

the role of scientists in answering managers’ questions about specific management options 

rather than in simply delivering technical presentations on their areas of expertise. 

Important refinements identified included improved impact estimates (e.g., power), and a 

more detailed treatment of uncertainty (e.g., humpback chub). 

Determination of next steps depends on a review by TWG and AMWG members. Some possible 

next steps that were suggested by workshop participants included: 

1. Options and Attributes Workshop. One day working session (project team and GCMRC staff 

and/or small subgroup of TWG) to refine management options and attributes. The key is to 

develop a strategic set of options that expose major differences in approaches to 

management (e.g., flow options, TCD option, SED options, and techno-fix options) rather 

than a long list of every possible combination of management options. 

2. Impact Estimation. Refined estimation of the impacts of the options on the attributes by 

the contractors. This will involve more accurate modeling or elicitation of expert 

judgments on attributes for which there is greater uncertainty or controversy. The 

uncertainty associated with each option must be better characterized. This will involve 

both a narrative and graphic summary of the range of possible outcomes (for holistic/direct 

ranking) and development of a summary statistic(s) to facilitate a weighting approach that 

accounts for uncertainty. 

3. Multi-attribute Trade-off Analysis Workshop. 2-3 day workshop to conduct a full-scale 

multi-attribute evaluation of options with TWG and/or AMWG members.