Defining Metrics for the Long-Term 

Experimental and Management Plan:

Concept, Plan, and FY21 Accomplishments

Annual Reporting Meeting

January 11, 2022

Helen C. Fairley

USGS Southwest Biological Science Center, 

Grand Canyon Monitoring and Research Center

U.S. Department of the Interior

U.S. Geological Survey
Presentation Outline

 Project overview

 What are metrics?

 Definitions

 Metrics as management tool

 Performance Metrics vs. Other Metrics

 Selecting performance metrics for LTEMP

 Project timeline and accomplishments

 Next Steps
Project Overview (Purpose and Need)

 LTEMP defines 11 Goals

 How do we know if Goals are being achieved?

 Need metrics!

 Section 6.1(c) of the LTEMP ROD* states:

“The DOI, in consultation with the AMWG, will develop 

monitoring metrics for the goals and objectives using those in 

Appendix C as a starting point.” 

(Note: Appendix C = performance metrics developed by Runge 

et al. (2016) to help select the preferred LTEMP alternative.)

 FY 21-23 TWP, Reclamation Project C.12

* Department of Interior, 2016, Record of Decision for the Glen Canyon Dam Long Term Experimental and 

Management Plan Final Environmental Impact Statement, December 2016. Bureau of Reclamation, Upper Colorado 

River Region, Salt Lake City, Utah and National Park Service, Intermountain Region, Lakewood, Colorado. 
What are metrics?
What are metrics?

 Metrics are measurements 

 Used to assess performance or condition

 “figures and data representative of an organization’s 

actions, abilities, and overall quality” (www.asq.org/qualityresources/metrics)

 “measures of quantitative assessments commonly used for 

assessing, comparing, or tracking performance or 

production” (From www.Investopedia.com)

 “quantifiable measures used to gauge performance or 

progress” (From www.grow.com)

 “numbers that tell you important information about a process 

under question.” (www.managementstudyguide.com)
Monitoring & Metrics

Monitoring serves a variety of useful purposes. 

Principal types and reasons for monitoring include:

 Effectiveness (or Performance) Monitoring

1. To assess effectiveness of policy, plan, or legislation

2. To evaluate progress towards achieving management 

objectives or regulatory standards

 Surveillance Monitoring

3. To detect incipient trends (“early warnings system”)

4. To determine resource status in order to decide 

appropriate management actions 

 Validation Monitoring

5. To increase our understanding of resource dynamics

6. To develop and refine models or predictions
LTEMP Metrics Project

 Focus is on defining metrics for 

Effectiveness Monitoring (“Phase 1”)

 “To assess effectiveness of policy . . .” and 

 “ track progress towards achieving management 

objectives . . .”

 LTEMP defines 11 specific goals

 How do we know if the goals are being achieved? 

Need to define performance metrics!

 Performance metrics tell us “achieved/ not achieved” -

- but not why

 NOT a Core Monitoring Plan 
LTEMP Goals (Fundamental Objectives)

 LTEMP Goals (from 2016 LTEMP FEIS and ROD)  1. Archaeological and Cultural Resources. Maintain the integrity of potentially affected NRHP-eligible or listed 

historic properties in place, where possible, with preservation methods employed on a site-specific basis. 

 2. Natural Processes. Restore, to the extent practicable, ecological patterns and processes within their range of 

natural variability, including the natural abundance, diversity, and genetic and ecological integrity of the plant 

and animal species native to those ecosystems. 

 3. Humpback Chub. Meet humpback chub recovery goals, including maintaining a self-sustaining population, 

spawning habitat, and aggregations in the Colorado River and its tributaries below the Glen Canyon Dam. 

 4. Hydropower and Energy. Maintain or increase Glen Canyon Dam electric energy generation, load following 

capability, and ramp rate capability, and minimize emissions and costs to the greatest extent practicable, 

consistent with improvement and long-term sustainability of downstream resources. 

 5. Other Native Fish. Maintain self-sustaining native fish species populations and their habitats in their natural 

ranges on the Colorado River and its tributaries. 

 6. Recreational Experience. Maintain and improve the quality of recreational experiences for the users of the 

Colorado River Ecosystem. Recreation includes, but is not limited to, flatwater and whitewater boating, river 

corridor camping, and angling in Glen Canyon. 

 7. Sediment. Increase and retain fine sediment volume, area, and distribution in the Glen, Marble, and Grand 

Canyon reaches above the elevation of the average base flow for ecological, cultural, and recreational purposes. 

 8. Tribal Resources. Maintain the diverse values and resources of traditionally associated Tribes along the 

Colorado River corridor through Glen, Marble, and Grand Canyons. 

 9. Rainbow Trout Fishery. Achieve a healthy high-quality recreational rainbow trout fishery in GCNRA and 

reduce or eliminate downstream trout migration consistent with NPS fish management and ESA compliance. 

 10. Nonnative Invasive Species. Minimize or reduce the presence and expansion of aquatic nonnative invasives. 

 11. Riparian Vegetation. Maintain native vegetation and wildlife habitat, in various stages of 

maturity, such that they are diverse, healthy, productive, self-sustaining, and ecologically 

appropriate.


Fundamental Objectives vs. Means

LTEMP Goals are Fundamental Objectives of LTEMP

Goal X: “Increase and maintain Resource X with y & z conditions“

actions important to goal X outcome 

Means Variable – e.g., intermediate state to achieving Goal X outcome

Means Variable Means variable, e.g., Benthic Conditions Means variable

Tributary inputs Dam releases (volumes, seasonality, magnitudes, etc.

Climate External Variables, e.g., Reservoir elevation Societal choices

Fundamental Objective: “Increase and maintain Resource X...”
LTEMP Goals = “fundamental objectives”

 “Fundamental objectives, as distinguished from 

means objectives, are objectives that are important in 

their own right—they are the desired outcomes of the 

decision, not because they lead to something else of 

importance but because of their inherent value.

Means objectives are pursued as a pathway to 

fundamental objectives but are not themselves of 

inherent value to the decision maker . . . .” 

Runge et al. 2015, p.5

 Fundamental objectives ≠ Means objectives 

 Fundamental obj. metrics ≠ Means obj. metrics
Example: HBC

“high juvenile survival rate of HBC is important as a means to achieving a 

sustainable population of HBC ... but juvenile survival is not itself the objective that 

the decision maker fundamentally cares about.” Runge et al. 2015, p. 5

Goal 3. Meet humpback chub recovery goals, including maintaining a selfsustaining population, spawning habitat, and aggregations in the Colorado River 

and its tributaries below the Glen Canyon Dam.

Meet HBC recovery goals

Juvenile survival rate

Habitat conditions Spawning success Predation success

LCR Floods

Gravel Exposure temperature

Climate

turbidity Predator pops..

Reservoir elevation Societal choices

Nutrients & Food

Dam releases (volumes, seasonality, magnitudes, etc.

Goal 3 Metrics:

1) No. adult HBC 

2) Stable/Upward 

trend in pop.
LTEMP Metrics Project: Objectives

 Determine technically feasible, sustainable metrics to 

provide objective measures of progress towards 

attaining the 11 LTEMP resource goals.

 Track the selected metrics to periodically evaluate 

and report status of 11 LTEMP goals relative to preLTEMP (pre-2016) condition and LTEMP projections.

 Improve annual/cyclical reporting by distilling large amounts of 

scientific information into a set of standardized metrics that can be 

communicated to DOI managers and GCDAMP stakeholders via 

standardized methods and graphics on a regular (annual?) basis.

 Inform other planning efforts (e.g., identify knowledge gaps).
LTEMP Metrics: Key Points

 Initial focus on LTEMP Fundamental Objectives

metrics

 metrics for other objectives & purposes come later

 Focus is on defining performance metrics for 

assessing & tracking achievement of LTEMP Goals 

(as described in the ROD)

 Performance Metrics measure how Goal X is doing, NOT 

why Goal X is increasing or decreasing

 Metrics for other purposes, e.g., “early warning” metrics, 

metrics for predictive models, TBD. 
Performance Metrics: Definitions

 Key Performance Indicators (KPI) 

“A performance indicator or key performance indicator 

(KPI) is a type of performance measurement [used to] 

evaluate the success of an organization or of a particular 

activity (such as projects, programs, products and other 

initiatives) in which it engages.” (from Wikipedia)

 Definition in Metric Plan

“objective, quantified measurements collected using 

standardized protocols that are indicative of resource 

condition so as to provide a measurement of how well the 

AMP is doing at attaining the established LTEMP goals”
Metrics design & selection

Some basic tenets of metrics design and 

selection (from metrics literature) include:

 Prioritize quality of metrics over quantity

 Design metrics that are easy to understand

 Design metrics that are easy to compare

 Prioritize metrics that are actionable*

* Depending on intended purpose of the metrics
LTEMP Performance Metrics: Criteria

 Reflects the expected performance outcome 

of each LTEMP goal or measures a 

performance outcome that triggers a legal 

compliance action or experiment

 Quantifiable (metric = measurable)

 Technically and financially feasible to 

measure

 Relevant to metrics used to select LTEMP 

preferred alternative
LTEMP Metrics Project: Initial Steps

 GCMRC scientists inventoried existing metrics 

in current monitoring projects 

 Existing metrics sorted into bins, e.g.,

 Is it well suited for tracking desired goal outcomes?

 Is it needed to trigger compliance/experimental actions?

 Is it a performance metric (i.e., “Metric of Success”) or a 

“means metric” (i.e., ecological system driver)? 

 Internal discussion re: suitability of specific 

metrics for tracking LTEMP goal outcomes 

 Does it directly measure a stated desired goal outcome?

 Technically feasible? Affordable? Other considerations? 
Project Timeline: FY21

 April-May 2021: Kickoff presentations to TWG and AMWG

 “Metric" defined for the purposes of this project.

 Project objectives identified; current inventory and assessment process 

and general timeline described.

 June 2021 TWG: 

 GCMRC presented draft process and timeline document for evaluating 

and selecting metrics, including draft criteria.

 General comments on definitions, criteria, etc. solicited from TWG

 July 2021:

 GCMRC and Reclamation brief Secretary’s Designee and other 

Department of the Interior managers regarding project and progress.

 General comments on objectives, process, and timeline solicited.

 August 2021 AMWG: 

 GCMRC presented revised draft process and timeline document for 

evaluating and selecting metrics, including draft criteria.

 General comments/ feedback on process, criteria solicited from AMWG

 July-August 2021: 

 GCMRC met with Tribes; began drafting metrics descriptions
FY21 Comments & Feedback Received

 During and following June TWG meeting

 Oral input (TWG chair, AZGF, GCWC, WAPA, NPS, 

Zuni, Navajo)

 Written input from GCWC, Zuni, Hualapai, State of 

CA, Anglers, WAPA

 Post-AMWG – no new comments received

 Post-October TWG – CREDA commented on draft 

hydropower metrics

 Post-Tribal w/ GCMRC meetings (7/14/21, 8/2/21) –

written comments received from Hopi, Navajo 
Summary of Stakeholder Comments

 Sounds good so far - please continue 

 Consider KA, existing plans as a source of metrics 

 Need to clarify goals before we can define metrics

 LTEMP Goals too general; need more explicit targets 

and some additional or different (new) goals

 Quantified metrics do not and can not adequately or 

appropriately reflect Tribal perspectives

 Tribes would like traditional knowledge to guide 

metrics selection

 Goal-specific metrics are contrary to Tribes’ interest 

in having a more holistic, integrated, ecosystembased approach to studying and managing Co. R.
FY22 Planned Project Timeline

 October 2021 TWG: 

 First draft of metric descriptions for 3-4 LTEMP goals 

 GCMRC presents draft LTEMP metrics (sample) to TWG

 GCMRC seeks input on metrics description format & how best 

to graphically represent metrics to meet stakeholder needs. 

 Additional Input from TWG due Oct. 31, 2021

 December 2021: 

 GCMRC pilots a sample of metrics in the FY2021 Annual 

Report, including sample graphics

 January 2022 (ARM and TWG):

 GCMRC solicits feedback from the TWG regarding the sample 

reporting of metrics in the FY2021 Annual Report. 

 February 2022:

 GCMRC solicits feedback from the AMWG and from DOI 

regarding the sample of metrics in the FY2021 Annual Report.
Next Steps

 Today: Seek additional TWG input (project focus, 

definitions, criteria, revised process & timeline)

 February-March 2022: internal (DOI) meetings

 April 2022: Present examples of performance 

metrics for 5-7 LTEMP goals to TWG

 June 2022: present draft metric descriptions (all 

goals) at June 2022 TWG 

 June-August 2022: GCMRC revises and finalizes 

draft metrics descriptions for all LTEMP goals 

 September 2022: final draft metric descriptions 

(all goals); to be presented at October 2022 TWG 
Questions?

July,1973 May 7, 2021 

Upper 103-Mile Camp, River mile 103.5, left bank