Science Consistency

Reviews

A Primer for Application

United States

Department of

Agriculture

Forest Service

FS-771

September 2003
James M. Guldin, David Cawrse, Russell Graham, Miles

Hemstrom, Linda Joyce, Steve Kessler, Ranotta McNair,

George Peterson, Charles G. Shaw, Peter Stine, Mark

Twery, and Jeffrey Walter

Guldin is project leader, Southern Research Station, U.S. Department of Agriculture Forest Service,

Hot Springs, AR; Cawrse is resources staff officer, Shoshone National Forest, Cody, WY; Graham is

research forester, Rocky Mountain Research Station, Moscow. ID; Hemstrom is research ecologist,

Pacific Northwest Research Station, Portland, OR; Joyce is project leader, Rocky Mountain Research

Station, Fort Collins, CO; Kessler is staff director for Wildlife, Fish, Rare Plants and Rangeland,

Region 9 Regional Office, Milwaukee, WI; McNair is forest supervisor, Idaho Panhandle National

Forest, Coeur d’Alene, ID; Peterson is project leader, Rocky Mountain Research Station, Fort

Collins, CO; Shaw is national program manager for pathology research, Office of the Deputy Chief

for Research, Washington, DC; Stine is research biogeographer and program manager, Pacific

Southwest Research Station, Sacramento, CA; Twery is project leader, Northeastern Research

Station, Burlington, VT; Walter is district ranger, Clackamas River Ranger District, Mt. Hood

National Forest, Estacada, OR.

Science Consistency

Reviews

A Primer for Application

United States

Department of

Agriculture

Forest Service

FS-771

September 2003
1

Introduction

Concern over Federal land management decisions has

grown in recent years. Public debates over activities

on Federal land have been contentious, especially

regarding management of national forests. Decisions

on the management and use of national forest lands

are based on many different considerations and values.

Although Federal land managers can make choices

concerning how to balance the various risks and

tradeoffs involved, their decisions face questions from

the public about whether the management direction

and its associated effects, outcomes, and outputs are

appropriate. 

The responsible official can use a number of methods

to evaluate the scientific information supporting those

decisions. Examples include scientific peer review, 

science advisory boards, science consistency reviews,

or expert opinion. 

This primer outlines a formal process for a science

consistency review (SCR). The SCR is defined as the

process used to determine whether an analysis or decision document is consistent with the best available science. That review is accomplished by judging whether

scientific information of appropriate content, rigor,

and applicability has been considered, evaluated, and

synthesized in the documents that underlie and that

implement land management decisions. 

These guidelines were developed to standardize the

SCR process and to guide land managers and others

who choose to employ it. Special attention was given

to ensuring the guidelines:

• Incorporated existing agency experience with science

consistency evaluations. 

• Can be used for small or large decisions, for situations

of little or great complexity, and with emphasis on 

practical application and real-world constraints.

• Were peer reviewed by scientists, land managers, and

administrators, both inside and outside the agency;

their comments were incorporated in the process.

• Were field tested, which also resulted in refinements

of the process.

Thus, the guidelines reflect current thinking by both

National Forest System (NFS) and Research and

Development (R&D) personnel on the best way to

implement the SCR.

Decisionmakers can use the SCR to evaluate whether

the draft analysis or decision document under review

has considered and correctly interpreted applicable

and available scientific information. In addition, the

SCR can help decisionmakers determine whether the

expected risks and consequences from the proposed

action and alternative actions are consistent with what

would be inferred from the scientific literature, and

whether they are correctly interpreted and disclosed 

in the draft document.

The product of the SCR is an interim or final report

from the review team to the responsible official that

states whether and how the elements being evaluated

met the basic evaluation criteria (see box 1 on p. 6).

Interim SCR reports are used to revise draft analysis

and decision documents. The reviewers then reexamine the revised documents. When this iterative process

is completed, the final SCR report documents that all

elements are consistent with current scientific information. The responsible official can then cite the final

SCR report as evidence that the information used to

make his or her decision is based on a document consistent with available scientific information 

The SCR is not intended to validate, ratify, endorse, 

or make the decision. Neither the SCR report nor the

reviewers who prepare it should make judgments about

the balance that managers should achieve among

competing objectives or what levels of risk they should

take. The SCR report does not advise a decisionmaker

for or against a particular course of action. The

responsible official remains solely responsible for the

decision.
2

SCR Participants and Their Roles

The responsible official is the Forest Service line officer

(regional forester, forest supervisor, or district ranger)

who is responsible for making and executing the decision

(table 1). The responsible official makes choices related

to selecting a preferred alternative, balancing risk, and

considering tradeoffs associated with ecological, social,

and economic outcomes and effects. Although the

responsible official may not be directly involved in the

SCR review itself, he or she is responsible for the

direction the draft document takes, how the feedback

from the review is considered and incorporated into

revisions of the draft document, and how the decision

is made.

Technical experts are those individuals participating

directly or indirectly in the development of draft analysis

or decision documents subject to review. Often technical

experts are interdisciplinary team members; they are

typically professional employees; and they may or may

not have advanced academic degrees. Generally, the

technical experts directly review, reference, and 

incorporate information from science literature, expert

opinion, or the science assessment, if applicable, into

the draft documents. They are also responsible for

modifying the draft document based on feedback in

the SCR report.

Science consistency reviewers are the experts who 

perform the science consistency review. A reviewer

must have scientific credibility in the field but not

necessarily an advanced academic degree. In some

cases a person with local or traditional knowledge 

may be appropriate. Reviewers can be R&D scientists;

university faculty members; scientists with State 

agencies, tribes, or other Federal agencies; or NFS

staff or technical experts not attached to the particular

project or unit under review. Reviewers can also be

scientists or experts from private companies, 

consultants, or nongovernmental organizations. Above

all, they must be knowledgeable, fair, and impartial.

The reviewers may act independently when working 

on small-scale efforts. For more extensive reviews,

they may need to have more interaction as a team,

and potentially may be led by a designated team leader.

The review administrator is responsible for the conduct

of the review and any interim or final SCR reports. The

review administrator may be a R&D station director, a

subordinate administrator, or a scientist designated by

the station director. The review administrator may also

be a regional office staff director, a staff director from

another national forest, or an appropriate research

administrator outside the Forest Service, such as a

dean or department head of a land grant university in

consultation with the agency. The review administrator

must be selected with sensitivity to the amount of 

controversy expected in the draft document, the desire

to maintain appropriate scientific independence from

the draft document, and the need to address concerns

that might arise over any perceived bias.

The review administrator identifies candidate reviewers,

selects reviewers and negotiates their availability and

schedules, disseminates the materials to be reviewed

to the review team, coordinates the review itself, and

addresses disagreements among different reviews or

reviewers. For a small-scale review, the review 

administrator may be one of the reviewers, working

either alone or as a member of a small team. For more

extensive reviews, the review administrator coordinates

and summarizes the reviews for the responsible official.

Regardless of the scale of the project, the review

administrator is responsible for preparing the SCR report

and transmitting that report to the responsible official.
3

Table 1. Roles of participants in an SCR, as the process unfolds.

Participant

Role

Identifies need for SCR1

. XX X2 X

Initiate the SCR. XX X

Establish scope of the SCR. Develop budget; identify sources XX XX

of funding. Develop timeline. Establish standards for 

interaction between participants and public. Establish 

criteria for selection of reviewers.

Conducts assessments and National Environmental Policy XX

Act analysis, develops biological evaluations, does cultural 

resource surveys, etc; develops alternatives and evaluates 

them; documents literature and information used for analyses;

conducts ID team meetings, prepares draft document.

Establish administrative structure for review team. X XX

Select reviewers.3

Develop elements to be evaluated. X XX X X

Approve elements to be evaluated. X XX

Convenes SCR team, distributes draft document XX

for review, administers interim/final SCR.

Performs the review. XX

Provides reviewers with requested documentation; XX

may develop additional documentation as needed.

Approves the interim/final SCR report. XX

Ensure communication between technical experts XX XX

and reviewers.

Ensure that interim or final SCR does not advocate X XX

any particular decision and that scientific information

and consistency evaluations are value neutral.

Reconcile reviews. X XX

Reviews and comments on interim/final SCR. XX X

Directs modification of document based on interim SCR. XX

Modify document based on interim SCR. X XX

Is responsible for and approves final SCR report. XX

Is responsible for management decision. XX

Responsible Technical Reviewer Review

Official Expert Administrator

XX = primary responsibility

X = secondary responsibility

1 SCR may be conducted on both analysis documents and

decision documents, such as drafts of the Environmental

Assessment or Environmental Impact Statement or the

Decision Notice or Record of Decision.

2 A technical expert may request the responsible official to

initiate an SCR if the technical expert believes that an unbiased

review is warranted.

3 There is a relationship between the selection of reviewers

and the selection of elements. Identification of additional 

elements during the process may necessitate additional or

different reviewers.

4 Modifications in the draft document may be subject to an

additional review by the reviewers (the iterative process).

Iterative steps for interim and final SCR reports4
4

Administration of the SCR

The responsible official decides whether to initiate the

SCR and is responsible for the documents under review.

Once the SCR is initiated, the review administrator is

responsible for the SCR process and reports, and the

responsible official is responsible for all modifications

of the documents under review. The SCR process is

most effective if agency officials work collaboratively

rather than confrontationally. 

Occasionally, a R&D administrator may be particularly

interested in the use of science in a draft analysis or

decision document that could be clarified using the

SCR process. This situation should be addressed 

collaboratively since it is normally the responsible 

official who initiates and funds the SCR. A R&D

administrator may initiate a technical or scientific

review of an analysis or decision document without

the collaboration or funding of the responsible official;

but in that event the responsible official reserves the

right to apply the SCR report or not, at his or her 

discretion. If strong disagreements cloud the decision

to engage or terminate the formal SCR process as

described, the next level of authority should resolve

the disagreements. 

Many factors enter into the decision to initiate an SCR

(table 2). The initial contact about whether to conduct

an SCR should be made between the responsible official

and the appropriate R&D scientist or administrator,

because station and region leadership should be aware

of any SCR within their respective jurisdictions. That

initial contact enables the responsible official to review

the various alternatives available for evaluating the

scientific information in the proposed decision and to

decide whether the SCR process is the best tool available

to support his or her decision. 

If the responsible official decides to engage the SCR

process, the review administrator becomes responsible

for its conduct and completion. Accordingly, the first

step is to identify the appropriate review administrator.

Then the responsible official initiates the review through

a formal written request to the review administrator.

Throughout the administration and conduct of the

SCR, the responsible official should maintain an

administrative record that documents the initiation,

conduct, and conclusion of the SCR.

Several factors are important to consider as the

responsible official and review administrator outline

the overall administrative context of the proposed SCR.

One suggestion to facilitate mutual understanding of

these issues is to prepare a formal written charter,

memorandum of understanding, or similar document

that states how these issues are to be handled for the

SCR under consideration. The charter must recognize

that the responsible official is responsible for the draft

document, but the review administrator is responsible

for the SCR report.

1. The first priority is to establish the scope of the 

review and the issues that are relevant to the 

review’s focus. The initial request from NFS deter

mines the general outline of the SCR, but the 

review administrator and responsible official must 

work together to decide exactly what form the SCR

will take.

2. The timeframe for the review must be established. 

The time commitment to complete such a process 

can be significant; the responsible official and 

review administrator must acknowledge that from 

the start.

3. The degree of openness of the SCR process among 

the participants and with the public should be 

established. The beginning of the process is the 

appropriate time to clarify these administrative 

relationships so as to minimize conflict.

4. The funding source for the SCR must be identified. 

Depending on the scope of the matter, the cost of 

conducting the SCR could be high. Generally, NFS 

would be responsible for the costs involved.

5. The review administrator selects the members of 

the review team, in consultation with the responsi

ble official. Reviewers should have three attributes:

expertise, credibility, and independence from 

authorship of the draft document under review. 

The number of reviewers on the team should relate

to the scale and magnitude of the issues and the 

available resources. 

6. The draft document under review and the support

ing materials must be introduced to the review 

team in an effective manner. Reviewers should be 

charged with examining the entire draft document 

with emphasis on their own individual specialties. 

The review administrator should generally resolve 

or address disagreements among reviewers. Team 

members should allocate group tasks among 

themselves or follow the review administrator’s 

direction. Discussions and assignments need to be

clear and focused.

7. The team should be directed to seek consensus as 

they conduct their review, but not given a mandate

to achieve it. The team members should be able to 

work independently in their areas of expertise; 

however, the group as a whole should be able to 

work together and provide input to evaluate each 

other’s reviews and ensure that issues raised are 

actually science consistency issues. Face-to-face 

meetings are occasionally valuable but are not 

required for the work to be accomplished.

There is no proper point in time to initiate an SCR,

but it is important to understand that the planning

process may proceed more smoothly if reviewers can

review draft documents, and technical experts refine

them, earlier rather than later in the planning process.
5

Table 2. A conceptual framework that the responsible official should consider in making the decision regarding

whether to conduct an SCR, the amount of effort to expend, and the time needed for an SCR. 

Potential Effects1 (Risks & Benefits)

Factor To Consider

Spatial and temporal scales • Localized site conditions • Broad geographic ranges and

• Small watershed multiple agencies

• Transcend organizational

boundaries

Duration of effects • Short-term effect on communities, • Long-term effect on communities,

economy, and/or environment economy, and/or environment

Scope of decision • Routine management actions • Large-scale regional and forest

(site-specific) plans or plan amendments

State of knowledge • Well-developed routine analysis • Emerging science and technology

• Professionally recognized science • Disputed findings and 

finding interpretations

Data availability • Well developed • Data gaps

• Generally accepted • Arguably insufficient data

• Associated risk small • Highly disputed

Scope of effects • Limited effect on or change to • Long-ranging associated risks to

communities the environment

Controversy • Limited in scope and action • Highly disputed and/or arguably 

insufficient data

Low (SCR probably not needed or High (SCR may be helpful and

need can be met by smaller effort) may require larger effort)

1 The term "effects" refers to risks, benefits, outcomes, and outputs.
6

The Review Process

To conduct the SCR, reviewers must read and comment

on the draft document and supporting materials. The

elements to be reviewed will depend on the particular

situation and will require coordination between the

review administrator and the responsible official. The

criteria to be met in the SCR do not depend on any

particular situation, and thus have been standardized

(box 1).

The review administrator, in consultation with the

responsible official, selects the elements to be reviewed

(box 2). Since not all elements subject to an SCR are

equally important, the responsible official and his or

her technical experts (the management side) should

collaborate with the review administrator and his or her

reviewers (the review team) to identify the appropriate

elements to be included. The management side brings

knowledge of policy significance, and the review team

brings knowledge of scientific and technical significance.

The review administrator needs to consider both of

these areas in order to develop a policy-relevant and

scientifically valid statement of those elements that

need an SCR. 

The process for conducting the SCR, outlined in box 3,

is designed to be flexible and scalable, especially for

smaller analysis or decision documents at the project

or plan amendment level. It is intended to make efficient

use of reviewer time and effort, analogous to that

invested in review of scientific manuscripts of equivalent

length. The iterative nature of the process means that

reviewers might be asked to review a document more

than once, but in subsequent reviews they must 

consider the entire revision of the draft document, not

just those elements that had been previously identified

as inconsistent with the evaluation criteria.

Box 1. Using Specific Criteria To Evaluate 

Consistency

Experience has led to the recommendation that

the elements should be reviewed with a standard

set of criteria. The following criteria should be used

to analyze each element relative to the document

under review and the decisions to be made:

1. Is the relevant scientific information considered?

2. Is the scientific information reasonably interpreted 

and accurately presented?

3. Are the uncertainties associated with the relevant 

scientific information acknowledged and documented?

4. Are the relevant management consequences identified

and documented, including associated risks and 

uncertainties?

Box 2. Selecting Elements To Review

Elements of the document for which science 

consistency review is desired or needed are difficult

to identify. The following selection factors may help

identify what issues are important to a particular

situation and whether the potential elements can

be addressed by a scientific review. Examples of

factors that indicate an element that should be

reviewed include one or more of the following. 

An element should meet both factors 1 and 2, 

plus at least one of factors 3 through 6, to merit

consideration in the review:

1. Scientific information about an element is 

important to the potential decision.

2. An element is significant enough to affect 

the substance of the decision.

3. Public interest or official policy has identified an 

issue as important. 

4. There is excessive conflict or uncertainty 

within the scientific community.

5. A potential element is relevant to the 

agency and within its decisionmaking authority. 

6. The information prepared for the decision 

document has sufficient scientific credibility.
7

Implementing the SCR Process

Widespread implementation of the SCR process within

the agency will require attention to several organizational

and policy issues. These issues include primary 

purpose, analysis gridlock, potential for appeals, 

and reviewer workload. Such issues point to the need

for guidance from agency leadership when the SCR

process is made available to the agency at large.

The SCR process proposed here will no doubt evolve

as it is applied in different situations. It is important

to adaptively respond to early experience with the SCR

process as outlined herein. For these and other reasons,

interim staff support at the Washington Office level

will be valuable as the SCR process is introduced to

the agency at large.

Initiating the review:

1. Initial discussions held between responsible official

and Forest Service R&D contact.

a. Explore alternatives for evaluating scientific 

information in the proposed decision.

b. Decide whether an SCR is the best tool to 

apply in the given case.

c. If responsible official elects to use the SCR 

process, select the review administrator.

2. Letter from responsible official to review 

administrator asks for review.

a. Prepare a written charter, MOU, or other 

authorizing document.

b. Establish the scope, content, review elements, 

and sideboards for the review.

c. Forecast the expected timeframe for the review.

d. Clarify the openness of the review among 

participants and with the public.

e. Determine how to pay for the SCR.

3. Review administrator establishes review team.

a. Consult with responsible official and other 

domain experts to find appropriate reviewers.

b. Establish administrative structure for the 

review team.

4. Review administrator and responsible official 

issue charge to the review team.

a. Introduce team charter, documents to be 

reviewed, and appropriate supporting materials.

b. Review administrator finalizes review team 

administrative structure and function.

Conducting the review:

1. Selecting the review elements. 

a. Technical experts provide reviewers with 

detailed initial list of elements from document.

b. Technical experts, reviewers, responsible official,

and review administrator review the proposed 

elements.

c. Revised list of elements and justifications are 

returned to review administrator for final 

approval.

Box 3. A Suggested Process for Conducting the Review

d. Elements are written to be answered with "yes"

or "no" under the evaluation criteria.

2. Scoring elements against criteria.

a. The elements and criteria form a matrix of 

rows and columns.

b. Each reviewer records a yes or no rating in 

each cell of the matrix of elements and criteria.

c. Ratings are accompanied by narrative justification

statements in which reviewers explain their 

evaluations and offer suggestions for changes 

that would achieve consistency.

3. Reconciling reviews.

a. Reviewers forward finished report to review 

team leader or review administrator.

b. Review administrator reconciles variations in 

review content and format.

c. Review administrator collates reviews and prepares

summary SCR report for responsible official.

4. Preparing the review report.

a. Draft or interim reports are anticipated as part

of an iterative process that may result in a 

revised document that needs an additional 

iteration.

b. Review administrator may conduct or request 

a review of the final team report to ensure that 

the report does not advocate any particular 

decision or alternative, and that the evaluations

are value neutral.

Completing the review:

1. Responsible official and review administrator 

jointly make the ultimate decision that the 

process is complete.

2. The interim or final report is transmitted from 

review administrator to responsible official.

3. Responsible official directs technical experts to 

refine draft document based on SCR report.

4. If necessary, revised document is returned to 

review team for further review.

5. Final SCR report and final decision are issued by 

review administrator and responsible official, 

respectively.
8

Table 3. A sample excerpt from a matrix of elements and criteria to be used in evaluating science consistency.

For each element, technical experts should provide an explanation of the purpose and thinking behind the element to improve the reviewer’s comprehension of the element to be rated. For each rating in each cell, reviewers

should provide a note explaining why and how the rating was derived, including suggestions that could be used

to improve a "no" rating.

Elements Criteria for Decision

1. Old Forests and Associated Species

1.a. Spotted owl habitat requirements 

and availability

1.b. Spotted owl population trends

1.c. Silvicultural prescriptions to create

old forest habitat 

1.d. Conservation approaches for 

Canada lynx

2. Fire and Fuels

2.a. Current and future fuel loading

2.b. Current and future wildfire trends

2.c. Smoke emissions and effects on 

human health

2.d. Effectiveness of treatments for fuel

reduction

3. Nonnative Invasive Species

3.a. Effects of spotted knapweed on 

native communities

3.b. Treatments to control spotted 

knapweed

3.c. Effects of zebra mussels on native 

aquatic communities

4. Economic and Social Well-Being

4.a. Rural community economic and 

population trends

4.b. Role of nontimber forest products 

in local livelihoods 

4.c. Recreational and economic 

importance of off-highway vehicles

4.d. Effects of timber harvest trends on

local communities

Is the relevant Is the scientific Are the uncertainties Are the relevant

scientific information associated with the management

information reasonably relevant scientific consequences

considered? interpreted and information identified and

accurately acknowledged documented,

presented? and documented? including 

associated risks 

and uncertainties?
9

Summary

The responsible official should consider the SCR as an

optional tool to use when he or she thinks that the

scientific information contained in a draft document

could benefit from a review. When conducted as 

suggested herein, the SCR will essentially serve as a

report to validate the scientific information used in the

decision, and it will enable the responsible official to

assert that a decision is consistent with available 

scientific information.

Acknowledgments

The authors thank Thomas J. Mills and Harv Forsgren

for their support in developing this paper on behalf of

the Forest Service National Leadership Team. Thanks

also to Meredith Webster, Brian Kent, Lisa Freedman,

Jack Troyer, Gary Pierson, and Dick Smythe for helpful

review comments.