TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

DRAFT FOR TWG REVIEW

October 2011

Appendix B

TWG Review and Approval of Individual Core Monitoring Plans

Despite potentially improving conservation decisions, the benefits of the information gained from costly 

monitoring programs may not always outweigh the costs of acquiring it. Understanding the nature of 

these costs and benefits is vital to evaluating them rationally in the context of financial limitations and the 

urgency of conservation issues (Field et al. 2007, Johnson et al. 2008, Legg and Nagy 2006). The 

GCDAMP is faced with a financial crisis, climate change, needed management actions, endangered 

species recovery, and the increasing costs of doing business has left the program with substantial 

limitations which affect our ability to do adaptive management. Monitoring is generally perceived as a 

rational and defensible activity in the pursuit of improved conservation outcomes. Rarely, however, do we 

critically assess the relative value of gaining information or the utility of what we collected (Field et al. 

2007). Legg and Nagy (2006) advise us that we must be prepared to forego monitoring in some cases by 

explicitly asking the question: is spending money on monitoring justified relative to funding other actions, 

including strategic research (or management actions)? Importantly, a decision to direct resources away 

from monitoring is not driven by reluctance to evaluate our conservation investments. Instead, this 

decision is driven by a desire to maximize expected conservation outcomes given limited resources (Legg 

and Nagy 2006).

1.0 Introduction

The GCMRC Monitoring and Research Plan (MRP2009; MRP) describes a four-step process for defining 

and refining core monitoring projects (1) develop a general core monitoring plan (CMP), (2) conduct 

information needs workshops with the Technical Work Group (TWG) in advance of convening 

independent protocol evaluation panel (PEP) reviews, (3) conduct PEPs for each resource goal, and (4) 

prepare final core monitoring program reports for each resource goal. The described processis is then 

followed by review and approval by TWG and AMWG. This appendix describes TWG’s role in the 

review and approval process and describes expectations by TWG when reviewing the individual plans.

Environmental decisions in the GCDAMP generally involve complex scientific and technical issues – in 

addition, a wide variety of program participants, value conflicts, scientific uncertainty, and social 

dynamics all combine to make environmental decisions difficult to make. The result of previous efforts to 

develop core monitoring and the CMINs themselves (Appendix A) represent a “Christmas tree” approach 

(described below) which includesd all the “ornaments” or questions posed by the diverse group within the 

GCDAMP. The current list of CMINs CMP waswere not adequately formulated with the recognition that 

there are trade-offs among budget allocation, precision of determining change in resource conditions, and 

risks that managers take when decisions are made based on results from the monitoring.

The goal of this CMP process is to further refine those monitoring choices using working in a 

collaborative relationship process between TWG, GCMRC, and AMWG. To accomplish this, the 

development of individual core monitoring plans will adopt a collaborative adaptive management 

frameworkprocess utilizing (a) program Goals and CMINs, (b) criteria for TWG evaluation of the 

element inclusion within plans, (c) trade-off analyses usingin a risk assessment framework, and most of 

all (d) stakeholder involvement in the development of all of these elements.
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

The GCMP identifies the general goals, objectives, scope, schedule, and funding level for each proposed 

core monitoring project as well as the program as a whole. The scope of the GCMP is based on the core 

monitoring information needs (CMINs) defined by AMWG in the 2003 Strategic Plan, as modified and 

prioritized by the 2005 Science Planning Group (Appendix A). The GCMP takes into account the 

feasibility of developing monitoring protocols to meet those needs while including a flexible approach for 

incorporating risk assessments and trade-off analyses to support decision making related to the scope and 

elements of the monitoring programs. The GCMP also identifies the process and strategies which will be 

used to develop and finalize individual core monitoring program plans. 

2.0 BackgroundHistory of Core Monitoring Development

As described in Chapter 1 of this plan, in March 2004, a core monitoring ad hoc committee composed of 

members of the GCD Technical Work Group and staff of the GCMRC was convened by a decision of the 

AMWG. This group of individuals, referred to as the Core Monitoring Team (CMT), represented a crosssection of many GCDAMP resource management concerns and scientific disciplines. Although the team 

was not successful in completing a CMP as was envisioned by AMWG (see Chapter 1 for further details), 

the CMT did explore some important policy-related issues and had some relatively clear conclusions 

about how the CMP should be developed. Unfortunately none of those conclusions were every truly 

tested at the policy-level (AMWG). The TWG would like to utilize these key conclusions by the CMT as 

a starting point for our guidance in this revised review and approval process.

On April 9, 2004, the CMT met and produced a position statement on core monitoring. The statement 

outlined a number of assumptions and resources to study. The CMT identified 10 assumptions that would 

need to be embraced in the development of the GCMP:

1. Use available technology, as appropriate,

2. Adopt a minimalist framework (e.g., no ornaments on the Christmas tree),

3. Meet the needs of stakeholders and answer their specific management questions,

4. Strive for automated techniques that are less invasive and more efficient,

5. The budget needs to support the plan (e.g., 40-60% of our budget for core),

6. Build for consistency,

7. Build for longevity,

8. Incorporate flexibility to adopt new technologies,

9. The plan will be reviewed and accepted by SAB/TWG/AMWG/GCMRC staff, and

10. The results of monitoring will be regularly reported.

In general the current approach by GCMRC follows these guidelines except for numbers 2 and 5 which 

relate to the scope and cost of the program. In various documents from the CMT, they refer to a goal of 

40-60% of the science budget to be used for core monitoring (not a percentage of the total budget) in 

order to allow for money to support compliance needs and experimentation – this was the impetus of the 

core monitoring concept idea was to describe the most important research which would occur in years 

when funds were not available to fund all of the projects we might wish to dolimited. On May 6, 2005 

GCMRC reiterated this goal in a memo from Helen Fairley to the TWG, it read; “It is anticipated that 40-

60% of the annual GCMRC budget will be devoted to projects within the FCMP [Final Core Monitoring 

Plan].” Given obvious budget constraints currently facing the program and likely to in the future due to 

issues such as climate change and others, the need to limit the core monitoring program is even more 

important than ever. The current GCMRC proposal represents about 75% of the science budget and 

highly constrains are our abilities to exercise adaptive management with current funding levels. This is 

nothing new, the CMT struggled with these issues during their work in 2004-2005 and came up with a 

series of criteria to evaluate what should be “core.” Thus, in this process we will be requesting 

developing tools to help us make these difficult choices such as a trade-off analyseis for each plan which 
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

incorporates different funding levels. The funding level generally proposed by GCMRC in this document 

general plan will be considered the “high” option; two other options below that level will also be 

requested by TWG. 

Another outcome of the April 9, 2004 meeting was a position statement articulating decisions, roles and 

responsibilities, definitions, and basic principles the group endorsed related to the process for how the 

CMP would be developed. The CMT reviewed and adopted the definition of core monitoring proposed in 

the AMP Strategic Plan:

Consistent, long-term, repeated measurements using scientifically accepted protocols to 

measure status and trends of key resources to answer specific questions. Core monitoring 

is implemented on a fixed schedule regardless of budget or other circumstances (for 

example, water year, experimental flows, temperature control, stocking strategy, 

nonnative control, etc.) affecting target resources.

Although there was clear recognition and acceptance that emphasis was to be placed on key resources and 

specific questions, CMT members eventually identified most resource categories evaluated in the EIS as 

being necessary components of the plan. A popular metaphor subsequently identified the plan as a 

“Christmas tree” on which everyone wanted to hang their favorite “ornaments.”

On May 4, 2004, the CMT was assembled again in an attempt to reduce the scope (that is, reduce the 

number of “ornaments”) of the CMP. Although the team strived for a minimalist strategy (i.e., no 

ornaments), this goal went largely unrealized. The CMT decided that the following resource categories of 

concern would be covered in the CMP: CMP: (1) sediment; (2) wildlife/vegetation; (3) fish; (4) food 

base; (5) register eligible historic properties; (6) other cultural resources of tribal concern; (7) hydrology; 

(8) water quality; (9) recreation; (10) threatened and endangered species; (11) power; and (12) non-native 

species. 

The CMT affirmed the importance of relating monitoring activities to the questions arising out of the 

AMP strategic plan. Relevant fundamental questions included the following: (1) What and why do 

managers need to know? (2) Where do they want to know it? (3) How frequently do they need to know? 

(4) What are the general methods to obtain this information? (5) What is the level of precision/accuracy 

needed? (6) How will the monitoring data be presented? (7) Is it answering the managers’ questions?, 

and (8) What are the metrics of success, and how is success defined?

The general consensus of those present was that the GCMRC had been given sufficient information to 

begin working on the plan, and GCMRC was instructed to do so without further delay. Incorporating the 

input from the CMT, a third draft of the CMP was distributed to the TWG on September 24, 2004, and 

was discussed at the March 10–11, 2005, meeting of the CMT. To help identify and document the 

potential shortcomings of projects included in the draft CMP, a list of criteria was developed that 

collectively addressed the concerns of CMT members for evaluating each project. An ad-hoc 

subcommittee was formed to develop a more refined process for addressing the evaluation criteria and to 

transmit that information to the CMT. Initially, at the March 10–11, 2005, meeting, the CMT developed 

seven criteria with which to evaluate how well each “green” project met an existing AMP goal. 

Evaluation criteria were compiled from CMT discussions, flip charts, and notes from individuals in 

attendance at the meeting. These seven criteria assumed that scientific methodology exists to accomplish 

all or part of the monitoring activity. Whether or not acceptable methodology exists was later added as 

criterion 8. The criteria were:

1. AMWG Priority: does the data directly or indirectly address an AMWG priority?
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

2. MOs and CMINs: does the data directly or indirectly address and existing MO and specific 

CMINs?

3. Compliance: does the data satisfy a specific legal compliance requirement outside of GCPA and 

the NPS Organic Act (e.g., EIS/ROD, ESA, LOR, NHPA)?

4. Legacy: Does the data contribute to an historical record that is important to continue?

5. Data quality/availability: is the accuracy and precision of the data known for proposed data 

collection, and if so, is the level of accuracy/precision adequate, inadequate, or more than 

adequate to meet the needs of the program?

6. Cost/benefit and risk assessment: what are the relative merits of collecting this data relative to 

other data collection?

7. Status of knowledge: what is the current status of data analysis and knowledge for the resource?

8. Metheodology: does methodology exist that provides acceptable accuracy, precision, and 

frequency of data?

On March 23, 2005, a conference call was held among the CMT subcommittee members and an outline of 

the process was developed. It was decided that a complete revision of the current CMP was impractical 

within the timeframes allowed in the current development schedule and that a provisional plan would 

have to suffice for 2005–06. Therefore, a fourth draft of the CMP, this time composed only of the “green 

projects,” was sent to the science advisors for review on April 5, 2005, and was distributed to the CMT on 

April 9, 2005. This fourth draft was discussed at the April 11–12, 2005, CMT meeting, where some CMT 

members expressed the opinion that the existing draft plan was fundamentally flawed in both format and 

content. General criticisms included the following:

1. The plan did not provide the level of detail desired by AMP members.

2. The plan lacked information on the accuracy, precision, and frequency needed to ascertain change 

over time.

3. The plan did not provide an ecosystem perspective.

4. The plan was not integrated between scientific disciplines.

5. There was continuing confusion over what is core monitoring vs. other kinds of monitoring.

6. There was confusion over whether yellow and red projects should be part of the plan or part of a 

different plan.

7. The plan did not provide enough detail as to how monitoring projects addressed AMP goals.

Some of these issues reflect the concerns of individual stakeholders, and others reflect group consensus. 

For example, the group agreed that the entire CMP needed to be reoriented around AMP goals and the 

data necessary to support those goals, rather than continuing along the path of attempting to evaluate the 

existing GCMRC monitoring program on a project-by-project basis. There was also group consensus that 

the CMP should address at least some aspect of all the resources identified in the AMP goals, minus goals 

3 and 12. In addition it was recognized that an essential component of the plan was still missing—an 

ecosystem perspective. Furthermore, the CMT determined that the plan was hampered by the lack of a 

clear definition by stakeholders of what core monitoring was trying to achieve within the AMP, including 

fundamental requirements such as how much change in a resource needed to be detected over what period 

of time. There was general agreement that these fundamental issues needed to be resolved through 

additional discussion among TWG members.

The CMT subsequently determined that even if existing projects met all the evaluation criteria, budget 

realities would continue to constrain AMP monitoring activities and that therefore, a process needed to be 

developed that prioritized AMP strategic goals, management objectives, and information needs so that 

rational decisions could be made relative to which monitoring activities would be funded at what levels 

and which ones would receive less funding or none at all. The CMT also decided that some additional 

criteria needed to be added, such as relative importance of a given resource for ecosystem function.
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

Once the CMT recognized that they needed to undertake an entirely new process for evaluating proposed 

monitoring activities, the group decided that GCMRC should complete a provisional plan for FY06 

quickly and with minimal additional effort so that the team could focus their full attention on developing 

an entirely new, AMP goal-oriented core monitoring plan. The resulting FY06 Provisional Core 

Monitoring Plan only included projects that had undergone one or more PEP evaluations, had been 

piloted and results peer-reviewed, and had been implemented for one to several years using methods 

deemed adequate for long-term monitoring. Projects in this category included (1) Lake Powell quality of 

water, (2) downstream surface water (discharge and stage measurements), (3) downstream quality of 

water for a limited suite of parameters, such as temperature, specific conductivity, and suspended 

sediment, (4) status of Lees Ferry rainbow trout, and (5) status of humpback chub in the Little Colorado 

River (LCR). These “green” projects constituted the core elements of the FY06 provisional draft plan.

Termination of the Core Monitoring Team approach to developing a core monitoring plan coincided with 

the initiation of the Science Planning Group (SPG) in the summer of 2005. The primary purpose of the 

SPG was to develop a long-term experimental plan, but in the course of this effort, several activities were 

undertaken that helped to further the aims of core monitoring plan development. Specifically, the SPG 

revisited, refined, and prioritized the resource goals and core monitoring information needs (CMINs) for 

the program. The results of this prioritization exercise, reflected in Appendix A, have been used to guide 

the development of this current plan.

3.0 TWG Role in the Development, Review, and Recommendation of Individual Plans

Theis current draft CMP describes the development of individual plans in detail and expands greatly on 

step 4 of the process (see Section 1.1) and the information necessary to support the development and 

implementation of individual core monitoring plans. The CMP outlines an adaptive management 

approach which integrates risk assessment, trade-off analyses, and specified criteria in the development of 

the individual plans such that managers will have the information necessary to make informed decisions 

about the scope, intensity, and cost of the core monitoring program. 

In 2004 the Core Monitoring Team indicated that an appropriate size for the monitoring program would 

be 40-60% of the GCDAMP budget. The current program is about 60% of the overall budget and about 

75% of the current GCMRC budget. The TWG has identified an initial goal of 40-60% of the science 

budget to be devoted to core monitoring. This initial goal was defined to allow for substantial funds to be 

available for research studies and other monitoring aspects designed to complement the core monitoring 

program in order to test whether the goals and objectives of the AMP are being achieved (i.e., ROD 

operations of GCD). 

The CMP results in projected expenditures above the goal articulated by the TWG based on myriad 

CMINs described by the program. GCMRC is not in a position to make policy choices about how to 

reduce costs, accept higher risk associated with potentially less robust programs, and define goals. Thus, 

this process must inherently be iterative working with the TWG and and AMWG G to develop a 

recommendation for individual plans that provide a range of options with varying tradeoffs of cost, 

statistical precision, and extent. GCMRC can articulate measures of risk associated with these choices, to 

the extent practicable, but the managers must make choices about how to implement the program. It is 

likely that the program will choose recommend less risk-averse options the first time through this process 

and that revisions are likely in order to weigh total core monitoring program costs against risk. The 

sediment program is a good example of this situation. The sediment core monitoring plan was 

conditionally approved by TWG knowing that after other plans were developed that we would want to 

revisit the sediment plan to reconsider costs and relationship to program goals and overall cost of core 

monitoring. Chapter 1 describes the first process undertaken by the Core Monitoring Team and the 
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

resulting “Christmas tree” approach. Thus, the cost estimates in this plan likely bracket the upper end of 

the possible funding needs. It is up to the managers to make informed choices about reducing the cost of 

this program based on the analyses described here.

3.0 Scope and Cost Evaluation of Individual Plans

The GCMP identifies the general goals, objectives, scope, schedule, and funding level for each proposed 

core monitoring project as well as the program as a whole. The scope of the CMP is based on the core 

monitoring information needs (CMINs) defined by AMWG in the 2003 Strategic Plan, as modified and 

prioritized by the 2005 Science Planning Group (Appendix A). The CMP takes into account the 

feasibility of developing monitoring protocols to meet those needs while including a flexible approach for 

incorporating risk assessments and trade-off analyses to support decision making related to the scope and 

elements of the monitoring programs. The CMP also identifies the process and strategies which will be 

used to develop and finalize individual core monitoring program plans. 

The CMP has been responsive to most of the higher priority CMINS, however given the scope of the 

information needs and funding limitations to develop monitoring programs, it does not currently account 

for all of them. Development and implementation of the core monitoring program for the GCDAMP 

would consume a large percentage of the current GCDAMP science budget based on the CMINs. The 

CMP includes initial estimates for costs and timeframes for program implementation but recognizes that a 

practical decision-making process will be needed by TWG to decide on a core monitoring program that 

meets stakeholder needs within available budget constraints. Those budget constraints cannot be 

articulated here as needs for management actions and other compliance needs are changing, but appear to 

be taking a larger percentage of the budget than in recent years.

In general, TWG is requesting that each individual plan contain 3 levels of funding within a trade-off 

analysis framework:

“High” – would implement the CMINs for that goal to the extent practicable and represent 

as close to full implementation as can be obtained with current resources, and is based on 

current implementation strategy by GCMRC.

“Medium” – would implement modest reductions in spending (about 10-30%) to implement 

the higher priority CMINs.

“Low” – would implement substantial reductions in spending (about 40-50%) to implement 

only the highest priority CMINs.

The analysis would not only show the reductions in cost, but the ability of the program to respond to 

CMINs (i.e., the ability to answer critical questions), and the rationale for those choices. These tradeoffs 

will be considered by TWG and a recommendation made to AMWG to consider the policy implications 

of those choices and to approve a plan that is both technically sound and well considered within our longterm financial limitations. This process would allow for a scientifically driven review of different funding 

scenarios and provide the decision-makers with the information necessary to make difficult policy 

decisions. It is inevitable that at current funding levels we cannot support robust monitoring capable of 

responding to all of the CMINs during the next 10 years, thus we understand that a serious review of our 

core monitoring program is critical in supporting the long-term funding needs of the GCDAMP. Given 

future funding needs by many competing activities, we expect that the monitoring program will be 

reduced to some extent, that process should include a structured review process (Appendix B) and include 

a clear understanding of the abilities which will be lost and which resources they would affect. In 

previous core monitoring discussion scope of the program was considered and from 2004 the Core 

Formatted: Font: Bold
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

Monitoring Team indicated that an appropriate size would be 40-60% of the GCDAMP budget. The 

current program is about 60% of the overall budget and about 75% of the current GCMRC budget.

Unresolved issue, how do we achieve and overall cost objective, how do we get to 40-60%?

4.03.1 Use of Structured Decision Making by TWGElements of Individual Core Monitoring Plans 

for Review by TWG

At the center of this process is the review by TWG and subsequent approval or disapproval based on 

various objective criteria proposed below. We propose that the TWG use a Structured Decision Making 

(SDM) process to evaluate the individual plans. Structured Decision Making is an organized approach to 

identifying and evaluating creative options and making choices in complex decision situations. SDM was 

successfully used to evaluate alternatives in the nonnative fish control plan EA with Reclamation (Runge 

et al. 2011). We propose a similar process here for each core monitoring plan in order to reach robust 

recommendations in a reasonable amount of time. Given how long this program has worked on a core 

monitoring plan it is clear that reaching decisions will not be easy and must be structured and allow for 

scrutiny of decisions.

Within those criteria we describe how GCRMC should generally meet those criteria and how TWG 

should score how well those criteria were satisfied. The framework of the individual plans described in 

section 3.0 above will go a long way in providing the policy information necessary to make good 

decisions, each program must be weighed for its significance in the adaptive management program. We 

propose that the TWG use a structured decision-making (SDM) process to evaluate these plans. This 

process would involve two TWG meetings with an initial and final review with a vote to recommend the 

plan for acceptance or rejection by AMWG. Structured Decision Making, or SDM, is an organized 

approach to identifying and evaluating creative options and making choices in complex decision 

situations. At the time of this writing SDM was being used to facilitate alternative development in the 

nonnative fish control plan EA with Reclamation. We propose a similar process here for each core 

monitoring plan in order to reach robust recommendations in a reasonable amount of time. Given how 

long this program has worked on a core monitoring plan it is clear that reaching decisions will not be easy 

and must be structured.

The SDM process is generally carried out in 6 steps (www.structureddecisionmaking.org 2010):

Step 1: Clarify the Decision Context

The first step in good decision making involves defining what question or problem is being 

addressed and why, identifying who needs to be involved and how, establishing scope and bounds 

for the decision, and clarifying the roles and responsibilities of the decision team. 

Step 1 is being carried out within this general plan, clarifying roles and decision 

processes.

More...

Step 2: Define Objectives and Evaluation Criteria

The core of SDM is a set of well defined objectives and evaluation criteria. Together they define 

"what matters" about the decision, drive the search for creative alternatives, and become the 

framework for comparing alternatives. 

Step 2 is carried out in part in Appendix B Section 3.3 which describes the TWG 

objectives for each individual plan. The individual evaluation criteria or attributes will 

Formatted: Keep with next, Keep lines

together

Formatted: Keep with next, Keep lines

together

Formatted: Indent: Left: 1", Keep with next,

Keep lines together

Formatted: Font: Italic

Formatted: Keep with next, Keep lines

together

Formatted: Indent: Left: 1"

Formatted: Font: Italic
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

be developed for each individual plan, tailored to that resource. These criteria were 

developed with the TWG using SDM processes. In addition, the primary objectives for the 

individual plans are the prioritized CMINs contained in Appendix A.More...

Step 3: Develop Alternatives

A range of creative policy or management alternatives designed to address the objectives is 

developed. Alternatives should reflect substantially different approaches to the problem or 

different priorities across objectives, and should present decision makers with real options and 

choices.

Step 3 is defined generally in Section 3.2 which describes a range of three alternatives 

based on cost which will be assessed in a trade-off analysis in each individual plan.

These alternatives should be distinguishable by the objectives described by TWG in 

Section 3.3.The three alternatives will be developed with TWG input by the GCMRC 

scientists. Alternatives will be developed which address the prioritized CMINs for that 

resource.More...

Step 4: Estimate Consequences

Step 4 is an analytical exercise in which the performance of each alternative is estimated in terms 

of the evaluation criteria developed in Step 2. Care must be taken to determine the focal areas of 

uncertainty and to ensure that these are represented properly in the analysis.

Step 4 is the heart of the analysis which will be conducted by GCMRC scientists and 

provided in the individual plans under the trade-off analysis.More...

Step 5: Evaluate Trade-Offs and Select

SDM is not a black box, and group discussion should always play a central role in evaluating 

preferences for alternatives. IHowever, in many cases, preference assessment techniques (such as 

swing weighting) may be used to help people understand their preferred alternativesalternative 

and reach agreement.

Step 5 will be carried out by TWG using facilitated SDM techniques to reach 

recommendation on which alternative or variation on those alternatives for each 

individual plan to recommend to AMWG. The key pieces of information used will be the 

criteria and attributes developed by TWG for each individual plan and the trade-off 

analysis of the alternatives provided by GCMRC.More...

Step 6: Implement and Monitor

The last step in the decision process then is to identify mechanisms for on-going monitoring to 

ensure accountability with respect to on-ground results, 

research to improve the information base for future 

decisions, and a review mechanism so that new 

information can be incorporated into future decisions.

Step 6 is being carried out within this general 

plan, clarifying monitoring and review processes.

Formatted: Font: Italic

Formatted: Indent: Left: 1"

Formatted: Font: Italic

Formatted: Font: Italic

Formatted: Font: Italic

Formatted: Font: Italic

Formatted: Indent: Left: 1"

Formatted: Font: Italic

Formatted: Indent: Left: 1"
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

This GCMP is expected to provides the information needed in Step 1 and to a certain extent Step 6 of the 

SDM process. We TWG will provide the objectives and attributes criteria for Step 2 in addition to the 

CMINs described in Appendix A. The range of below in the 10 criteria listed and described. We provide 

the range of alternatives for Step 3 ins described in this plan section 3.0 above; i.e., , requesting “high”, 

“medium”, and “low” scenarios. TWG would employ a We then propose that a series of two workshops 

(initial and final review) would occur at TWG to fully implement the evaluation Step 5process and 

focusing on SDM Steps 4-6. The specifics of those workshops and the SDM process would need to be 

worked out at a later date based on the complexity of the core monitoring plan and the funding and 

availability of experts to conduct the SDM. TWG should be involved in determining the appropriate SDM 

techniques used to reach a recommendationprocess.

3.2 Trade-off Analysis Using Three Alternatives

The CMP has been responsive to most of the higher priority CMINs, however given the scope of the 

information needs and funding limitations to develop monitoring programs, it does not currently account 

for all of them. Development and implementation of the core monitoring program for the GCDAMP 

would consume a large percentage of the current GCDAMP science budget based on the large list of 

CMINs. The CMP includes initial estimates for costs and timeframes for program implementation but 

recognizes that a practical decision-making process will be needed by TWG to decide on a core 

monitoring program that meets stakeholder needs within available budget constraints. Those budget 

constraints cannot be articulated here as needs for management actions and other compliance needs are 

changing, but appear to be taking a larger percentage of the budget than in recent years.

TWG is requesting that each individual core monitoring plan contain 3 levels of funding within a tradeoff analysis framework:

“High” – would implement the CMINs for that goal to the extent practicable and represent 

as close to full implementation as can be obtained with current resources, and is based on 

current implementation strategy by GCMRC.

“Medium” – would implement modest reductions in spending (about 10-30%) to implement 

the higher priority CMINs.

“Low” – would implement substantial reductions in spending (about 40-50%) to implement 

only the highest priority CMINs.

The analysis would not only show the reductions in cost, but the ability of the program to respond to 

CMINs (i.e., the ability to answer critical questions), and the rationale for those choices. These tradeoffs 

will be considered by TWG and a recommendation made to AMWG to consider the policy implications 

of those choices and to approve a plan that is both technically sound and well considered within our longterm financial limitations. This process would allow for a scientifically driven review of different funding 

scenarios and provide the decision-makers with the information necessary to make difficult policy 

decisions. It is inevitable that at current funding levels we cannot support robust monitoring capable of 

responding to all of the CMINs during the next 10 years, thus we understand that a serious review of our 

core monitoring program is critical in supporting the long-term funding needs of the GCDAMP. Given 

future funding needs by many competing activities, we expect that the monitoring program will be 

reduced to some extent. That process should include a structured review process and include a clear 

understanding of the abilities which will be lost and which resources they would affect.

3.3 TWG Objectives and Evaluation Criteria (or Attributes)
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

The core of SDM is a set of well defined objectives and evaluation criteria. Together they define "what 

matters" about the decision, drive the search for creative alternatives, and become the framework for 

comparing alternatives. In SDM, evaluation criteria (or attributes) are used to characterize the degree to 

which different alternatives are expected to meet objectives. They are used to: 

• compare alternatives accurately and consistently; 

• expose trade-offs including trade-offs among different degrees of uncertainty; 

• generate productive discussion about better alternatives; 

• prioritize information needs; 

• communicate the rationale for and improve the transparency of decisions. 

It isn't easy to define good evaluation criteria that are widely agreed upon by stakeholders, experts and 

decision makers. However, the up-front investment pays off in streamlined decision making, for two 

principal reasons: 

• because data, modeling and expert judgment processes are focused on producing decisionrelevant information; 

• because large numbers of very complex options can be consistently and efficiently evaluated by 

multiple decision makers. 

TWG held a workshop on March 9, 2011 to develop the general list of evaluation criteria using a 

consensus building workshop method. This list would then be refined and criteria (or attributes) would be 

developed specifically for each individual plan by TWG prior to the development of the alternatives and 

trade-off analysis. TWG generally considered two questions when developing the objectives:

What specific metric could we use to report the impact of these alternatives (High, Medium, 

Low) on this objective (CMIN)?

What specific information would you like to see to be able to evaluate the impact of these 

alternatives on this objective?

Using the Consensus Building Workshop method, TWG developed the following nine objectives and 

criteria for each objective. As discussed above, before the development of each individual plan, TWG will 

need to revisit and refine this list and describe attributes (metrics) that TWG can use to assess the 

differences in meeting these objectives across the three alternatives. Also, the primary objectives which 

define the alternatives are the CMINs themselves. How well the individual monitoring program meets 

those objectives and all of the other goals will be assessed by TWG using these criteria below.

CMINs

Addresses the needs identified for each resource within the prioritized CMINs (Appendix A)

Uses the prioritization of CMINs to determine which can be funded under each alternative

Note whether or not they are being addressed directly or indirectly

Specify the information needed to answer the CMINs completely

Describe what would be needed to overcome current limitations in answering the question completely

COMPLIANCE WITH LAW AND POLICY

Compliance (Federal/state/tribal)

Compliance – ESA, NHPA, Law of River, EO13007, GCPA, NPS Organic Act, EIS/ROD

Consistency with park values and no impairment

Degree of linkage to dam operations

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Indent: Left: 0.5"

Formatted: Font: Not Bold

Formatted: Font: Not Bold

Formatted: Font: Bold

Formatted: No underline

Formatted: Font: Bold
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

Related to GC Dam Operations

Meets mandates of GCPA

Comply with laws and regulations

Meets compliance needs (LOR, NEPA, etc...)

Meets legal goals/objectives

Related to “dam operations and other actions” (GCPA)

DATA QUALITY

Geospatially and temporarily related datasets

Objectivity and replication of methods, measurements, and assessment

Appropriate scientific methods (controls, sample, study design)

Use of best science/technology/TEK

Data provide a measure of confidence/reliability

Result in measurable outcomes

Adequacy of existing knowledge

Robust metric for determining effects

UTILITY OF DATA

Data are useful and timely

Timeliness of information to make management decisions

In a risk analysis, does it contribute in relation to its cost?

Ability to detect threats (contingency/risk assessment for speices)

Flexibility in periodicity of monitoring

Relevance of measured indicators to dam operations

Dual role of data and use in model generation

Relevance to rigorous, defensible predictive model

Adaptable/flexibility of timing and frequency

Does the data contribute to an historical record that is important to continue (legacy data)?

Are the data accurate and precise to meet management objectives? (e.g., are study design protocols 

sufficient to meet management goal to have 80% power (beta = .20) to detect a 20% change in the 

resource trajectory over 10 years while maintaining a 10% false change error rate (alpha = 0.10)? Or a 

similar management goal based on the resource?)

Ability to detect trends, adequate statistical power demonstrated for specified scenarios

Metrics and indicators appropriate to CMINs

MINIMIZE ADVERSE IMPACTS

Impact to visitor experience

Geographic and demographic extent of effects

Potential impact on resource being monitored

Minimal impacts to CRE

Impacts to Tribal Trust resources

Considers negative impacts on other resources

INTEGRATION OF CULTURAL AND SOCIAL VALUES

Considers environmental and cultural values

Zuni is happy

Considers economic values 

Sensitive to tribal concerns?

Stakeholder interest (i.e., AMG, TWG, public)

Satisfies tribal and public trust responsibilities

Considers social values

Formatted: No underline

Formatted: Font: Bold

Formatted: No underline

Formatted: Font: Bold

Formatted: No underline

Formatted: Font: Bold

Formatted: No underline

Formatted: Font: Bold
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

Tribal monitoring results can be meaningfully integrated

Culturally relevant and sensitive to cultural concerns

ADDRESSES GOALS AND PRIORITIES

Meets hierarchy of CMIN priorities, indirectly or directly

Does it relate to AMP goals 1-12 in strategic plan?

Meets program priorities including DOI priorities

Does it support (data linked) understanding and attainment of DFCs?

Does it support (data linked) decisions to make management actions?

Scope of support for CMINs described (full or partial)

Appropriate to resource goals (DFCs) and CMINs

INFORMATION MANAGEMENT

Adequacy of information management

Methodologies described and reasonable given other systems

ECOSYSTEM APPROACH

Is it directed to keystone resources, does the data represent the resource or ecosystem?

Prioritized by trophic structure

Tie-in to ecosystem model

Integration into big picture 

Provides for multiple needs

Do the data relate specifically to a critical ecosystem function?

Can it be used as an indicator of overall ecosystem health? 

Can it represent multiple resources at equivalent or lower trophic levels?

COST EFFECTIVENESS AND AFFORDABILITY

Maximize benefit/cost

Cost can be accommodated by program

Affordability and budget implications (cost)

Integration with other monitoring programs

No duplication of effort

Collaborative funding sources

1. AMWG Priority: does the data directly or indirectly address an AMWG priority?

a. GCMRC: 

• Describe the relationship to AMWG priorities and DFCs

• Discuss whether or not an AMWG priority is being addressed directly or indirectly

• If the priority is not being addressed completely, describe the scope of what is and is 

not being addressed by current data.

b. TWG: 

• do the activities relate to the 2004 AMWG priorities and draft DFCs (if available) to 

determine if it is a priority? 

2. MOs and CMINs: does the data directly or indirectly address an existing MO and specific CMINs?

Formatted: No underline

Formatted: Font: Bold

Formatted: Font: Bold

Formatted: Font: Bold

Formatted: No underline

Formatted: Font: Bold
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

a. GCRMC:

• Each individual plan must describe the CMINs that it relates to, and how the research activity will 

satisfy those CMINs in order of priority.

• Note whether or not they are being addressed directly or indirectly.

• Specify the information needed to answer the CMINs completely.

• Describe what would be needed to overcome current limitations in answering the question 

completely.

b. TWG: 

• Does the program address higher priority CMINs? Lower priority CMINs may become MINs as part 

of the TWG recommendation using the cost/benefit analysis below (i.e., High, Medium, Low funding 

scenarios).

3. Compliance: does the data satisfy a specific legal compliance requirement of GCPA, NPS Organic 

Act, EIS/ROD, ESA, LOR, or NHPA?

a. GCMRC: 

• Describe the compliance requirements to the extent known by GCMRC, and how/why the data 

address them.

• If the compliance requirement is not being addressed completely, the scope of the what is and is not 

being addressed.

b. TWG: 

• does the data satisfy a specific legal compliance requirement of GCPA, NPS Organic Act, EIS/ROD, 

ESA, LOR, or NHPA?

4. Legacy: Does the data contribute to an historical record that is important to continue?

a. GCMRC:

• Provide a complete description of the legacy data to include original purpose, where collected, 

accuracy/precision/frequency of data, was it validated by an acceptable QC/QA plan, PEP review, 

metadata available, does the data meet GCMRC standards?

b. TWG:

• Does the data contribute to an historical record that is important to continue?

• Could the data be continued in a less costly scenario using “bridge” sampling to ensure statistical 

robustness of the change? Does the analysis take these options into account?

5. Ecosystem importance: how well does the CMIN that is being responded to in the activity represent 

the resource or ecosystem?

a. GCMRC:

• How well do the data represent the resource and the ecosystem?

• Do the data relate specifically to a critical ecosystem function?

b. TWG: 

• Can it be used as an indicator of overall ecosystem health? 

• Can it represent multiple resources at equivalent or lower trophic levels?

6. Data quality/availability: is the accuracy and precision of the data known for proposed data 

collection, and if so, is the level of accuracy/precision adequate, inadequate, or more than adequate to 

meet the needs of the program? What level of statistical power is available over 10 years based on a 

false change error (alpha) of .05 to .1?

a. GCMRC:

• What data is being proposed for collection?

• How will the data be collected?

• Where will the data be collected?

Formatted

Formatted

Formatted

Formatted: Indent: Left: 0.25", No bullets or

numbering
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

• Describe the accuracy/precision/frequency necessary for the program to meet the management 

objectives?

• What is the statistical power (beta), and the objective over what time period?

• Describe the data QC and QA plan?

• Specify whether or not GCMRC compliant metadata has been developed.

• Specify whether or not the data meet GCMRC data standards.

• How and where will the data be stored and shared with the program and cooperators?

b. TWG:

• Are the data accurate and precise to meet management objectives?

• Are study design protocols sufficient to meet management goal to have 80% power (beta = .20) to 

detect a 20% change in the resource trajectory over 10 years while maintaining a 10% false change 

error rate (alpha = 0.10)? Or a similar management goal based on the resource?

7. Cost/benefit and risk assessment: what are the relative merits of collecting this data relative to 

other data collection?

a. GCMRC:

• Describe the consequence of collecting the data with higher and lower accuracy, precision, and 

frequency in terms of cost, ability to detect change over time, knowledge, and risk to the resource. 

Use the “high”, “medium”, and “low” trade-off analysis framework described above in section 3.0 to 

characterize these various levels of funding and analysis.

• Describe how the data are being or will be analyzed and what types of publications or other 

products are likely as a result.

•

b. TWG:

• What changes to the resource will the accuracy, precision, and frequency of data collection pick 

up (climatic, natural variability, natural or man-made events, etc.) over what minimum time interval?

• What are the implications to adaptive management for the trade-off between cost of the various 

program and the likelihood of successful adaptive management actions?

8. Status of knowledge: what is the current status of data analysis and knowledge for the resource

that the data is intended to monitor?

a. GCMRC:

• Discuss whether or not the data has been analyzed.

• Discuss whether or not targets and/or metrics have been established.

• Describe what the data shows relative to AMWG priorities and CMINs.

• Discuss whether or not the data are measuring the right things/variables.

• Discuss the current status and trend of the resource based upon this data.

• Describe what is known about cause and effect relationships, and relevance of data to evaluating 

such relationships.

• Discuss whether or not the data is meeting the AMP’s need relative to detecting decline or 

improvement of a resource over a useful period of time.

b. TWG:

• Evaluate the robustness of the data based on the variables above. 

9. Methedology: does methodology exist that provides acceptable accuracy, precision, and 

frequency of data?

a. GCMRC:

• Describe the proposed methodology.

Formatted: Indent: Left: 0.25"

Formatted: Indent: Left: 0.25", No bullets or

numbering

Formatted: Indent: Left: 0.25"

Formatted: Indent: Left: 0.25", No bullets or

numbering

Formatted: Indent: Left: 0.25"

Formatted: Indent: Left: 0.25", No bullets or

numbering
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

• Is similar work being done elsewhere? If yes, provide a synopsis of similar work being done and 

methodology used.

• Describe possible alternative methodologies, and discuss the pros and cons of these alternative 

methods in terms of costs, accuracy, precision, and frequency of data.

b. TWG:

• Evaluate the methodology, does it meet management needs at the least cost?

10. Trade-off Analysis: weigh criteria above with three cost scenarios and assess risk and benefits to the 

program considering 

a. GCMRC: for each plan provide three scenarios for funding and implementation

• “High” – would implement the CMINs for that goal to the extent practicable and 

represent as close to full implementation as can be obtained with current resources, 

and is based on current implementation strategy by GCMRC.

• “Medium” – would implement modest reductions in spending (about 10-30%) to 

implement the higher priority CMINs.

• “Low” – would implement substantial reductions in spending (about 40-50%) to 

implement only the highest priority CMINs.

b. TWG: evaluate the options to determine which best meets the programs overall needs at the 

lowest cost. Are the objectives met using variables which represent the change of interest 

over the desired time period to elicit management actions? Consider all variables above in the 

SDM framework, which option provides the greatest overall gain or value to the program?

The primary purpose of the SPG process was to develop a long-term experimental plan, but in the course 

of this effort the SPG revisited, refined, and prioritized the resource goals and core monitoring 

information needs for the program. The results of this prioritization exercise, reflected in Appendix A, 

have been used to guide the development of the monitoring protocols for the individual core monitoring 

plans. The priorities and ranking provided by the SPG will be used as the primary criteria in developing 

the individual plans. The SPG also recognized the need for the GCDAMP to more explicitly define the 

goals of the AMP and desired future conditions. At the time of this writing the AMP has begun serious 

consideration of desired future conditions. However, without DFCs it is impossible to determine which of 

the core monitoring proposals meets program needs, or perhaps is beyond our needs. At the December 1, 

2009 Core Monitoring Workshop with TWG, the second highest concern was whether we can proceed 

without DFCs, and if so how and what does it mean to the program? Many of the program elements may 

not need DFCs to be approved, but others might really need them, how do we move forward? This 

question is unresolved except to say that this program can move forward in developing the individual 

plans with good trade-off analyses and risk assessments which will likely have to be revisited in an 

adaptive management framework after DFCs are established. At that point the plans can be modified to be 

more responsive to the DFCs. It is likely that without DFCs the plans will have to be broader in scope, 

and thus more costly to implement.

CitationsCITATIONS

Legg, C.J. and Nagy, L. (2006) Why most conservation monitoring is, but need not be, a waste of time. J. 

Environ. Manage. 78, 194–199

Runge, M.C., Bean, Ellen, Smith, D.R., and Kokos, Sonja, 2011, Non-native fish control below Glen 

Canyon Dam—Report from a structured decision-making project: U.S. Geological Survey Open-File 

Report 2011–1012, 74 p.,

Formatted: Indent: Left: 0.25"
TWG Review and Approval of Individual Core Monitoring PlansGeneral Core Monitoring Plan, Appendix B October 7, 2011

Sarah E. Johnson, E. L. Mudrak, E. A. Beever, S. Sanders, and D. M. Waller. 2008. Comparing power 

among three sampling methods for monitoring forest vegetation. Can. J. For. Res. 38(1): 143–156.

Scott A. Field, Patrick J. O’Connor, Andrew J. Tyre, and Hugh P. Possingham. 2007. Making monitoring 

meaningful. Austral Ecology (32); 485–491