Disputes over Science and Dispute Resolution 

Approaches—A Survey of Bureau of 

Reclamation Employees

Open-File Report 2012–1186

U.S. Department of the Interior

U.S. Geological Survey
ii

U.S. Department of the Interior

KEN SALAZAR, Secretary

U.S. Geological Survey

Marcia K. McNutt, Director

U.S. Geological Survey, Reston, Virginia: 2012 

For more information on the USGS—the Federal source for science about the Earth,

its natural and living resources, natural hazards, and the environment—visit 

http://www.usgs.gov or call 1–888–ASK–USGS

For an overview of USGS information products, including maps, imagery, and publications,

visit http://www.usgs.gov/pubprod

To order this and other USGS information products, visit http://store.usgs.gov

Suggested citation:

Burkardt, Nina, and Ruell, E.W., 2012, Disputes over science and dispute resolution approaches— 

A survey of Bureau of Reclamation employees: U.S. Geological Survey Open-File Report 2012–1186, 

49 p.

Any use of trade, product, or firm names is for descriptive purposes only and does not imply 

endorsement by the U.S. Government. 

Although this report is in the public domain, permission must be secured from the individual 

copyright owners to reproduce any copyrighted material contained within this report.

The findings and conclusions in this report are those of the author(s) and do not necessarily

represent the views of the Bureau of Reclamation. 
iii

Contents

Executive Summary....................................................................................................................................................... 1

Main Findings............................................................................................................................................................. 2

Background ................................................................................................................................................................... 4

Characteristics of Disputes over Science................................................................................................................... 4

Issues Involved....................................................................................................................................................... 4

Parties Involved...................................................................................................................................................... 4

The Science under Dispute .................................................................................................................................... 5

Dispute Resolution Methods................................................................................................................................... 5

Defining Dispute Resolution ................................................................................................................................... 6

Web Survey................................................................................................................................................................... 7

Topics of Survey Question Units................................................................................................................................ 7

Survey Launch........................................................................................................................................................... 8

Study Objectives and Results........................................................................................................................................ 9

Respondents’ Professional Background and Experience........................................................................................... 9

Objectives and Hypotheses.................................................................................................................................... 9

Summary of Findings ............................................................................................................................................. 9

Detailed Results ....................................................................................................................................................10

Involvement in Disputes over Science ......................................................................................................................11

Objectives and Hypotheses...................................................................................................................................11

Summary of Findings ............................................................................................................................................11

Detailed Results ....................................................................................................................................................11

Dispute Characteristics .............................................................................................................................................12

Objectives and Hypotheses...................................................................................................................................12

Summary of Findings ............................................................................................................................................12

Detailed Results ....................................................................................................................................................13

Dispute Resolution Methods .....................................................................................................................................14

Objectives and Hypotheses...................................................................................................................................14

Summary of Findings ............................................................................................................................................14

Detailed Results ....................................................................................................................................................14

Number of Issue and Party Types Involved in Dispute Versus Method Types Used.................................................15

Objectives and Hypotheses...................................................................................................................................15

Summary of Findings ............................................................................................................................................16

Detailed Results ....................................................................................................................................................16

Contested Factors in Dispute over Science ..............................................................................................................17

Objectives and Hypotheses...................................................................................................................................17

Summary of Findings ............................................................................................................................................17

Detailed Results ....................................................................................................................................................17

Evaluations of Performance of Dispute Resolution Method Types ...........................................................................17

Objectives and Hypotheses...................................................................................................................................17

Summary of Findings ............................................................................................................................................18

Detailed Results ....................................................................................................................................................19

Dispute Resolution Professionals (DRPs).................................................................................................................20

Objectives and Hypotheses...................................................................................................................................20

Summary of Findings ............................................................................................................................................21

Detailed Results ....................................................................................................................................................21

Outcomes of Disputes over Science .........................................................................................................................22
iv

Objectives and Hypotheses...................................................................................................................................22

Summary of Findings ............................................................................................................................................23

Detailed Results ....................................................................................................................................................23

Resolved Disputes over Science ..............................................................................................................................24

Objectives and Hypotheses...................................................................................................................................24

Summary of Findings ............................................................................................................................................24

Detailed Results ....................................................................................................................................................24

Outcomes of Decisions Hindered by Disputes over Science ....................................................................................25

Objectives and Hypotheses...................................................................................................................................25

Summary of Findings ............................................................................................................................................25

Detailed Results ....................................................................................................................................................26

Experiences with Multiple Disputes over Science and (or) Dispute Resolution Method Types .................................27

Objectives and Hypotheses...................................................................................................................................27

Summary of Findings ............................................................................................................................................27

Detailed Results ....................................................................................................................................................28

Evaluations of Dispute Resolution Resources for Reclamation Employees..............................................................28

Objectives and Hypotheses...................................................................................................................................28

Summary of Findings ............................................................................................................................................28

Detailed Results ....................................................................................................................................................28

Acknowledgments ........................................................................................................................................................29

References Cited..........................................................................................................................................................29

Appendixes...................................................................................................................................................................33

Appendix 1................................................................................................................................................................34

Web Survey Questions..........................................................................................................................................34

Appendix 2................................................................................................................................................................43

Analyses of Survey Responses.............................................................................................................................43

Respondents’ Professional Background and Experience......................................................................................43

Involvement in Disputes over Science...................................................................................................................43

Dispute Characteristics .........................................................................................................................................44

Dispute Resolution Methods..................................................................................................................................44

Number of Issue and Party Types Involved in Dispute Versus Method Types Used.............................................44

Contested Factors in Disputes over Science.........................................................................................................45

Evaluations of Performance of Dispute Resolution Method Types........................................................................45

Dispute Resolution Professionals (DRPs) .............................................................................................................45

Outcomes of Disputes over Science .....................................................................................................................46

Resolved Disputes over Science...........................................................................................................................46

Outcomes of Decisions Hindered by Disputes over Science.................................................................................46

Experiences with Multiple Disputes over Science and (or) Dispute Resolution Methods ......................................47

Evaluations of Dispute Resolution Resources for Reclamation Employees ..........................................................47

Appendix 3................................................................................................................................................................48

Glossary of Analyses.............................................................................................................................................48

Figures 

1. Proportion of respondents that selected each of the dispute resolution method types ................................... 15

2. Means and 95-percent confidence intervals for the number of non-Reclamation party types involved 

in disputes over science within each of the dispute resolution method categories ......................................... 16

3. Means and 95-percent confidence intervals (of the summated method performance scale for each of 

the dispute resolution method categories ....................................................................................................... 20
v

4. How respondents rated the perceived effects of dispute resolution professionals on dispute resolution 

processes on a 5-point “negative effect” to “positive effect” scale .................................................................. 22

5. Proportions and sample sizes of responses in each of the dispute outcome categories ................................ 23

6. Proportions and sample sizes of responses in each of the decision outcome categories............................... 26

7. Proportions and sample sizes for each of the categories of dispute outcomes within each of the 

decision outcome categories .......................................................................................................................... 27

Tables

1. Categories of dispute resolution methods defined by what parties and what activities they involve................. 6

2. Sample sizes and proportions of responses within each of the professional background and experience 

variables categories........................................................................................................................................ 10

3. Sample sizes and proportion of responses that indicated that each scientific issue type had been 

involved in the dispute over science ............................................................................................................... 13

4. Sample sizes and proportions of responses that indicated that each type of party was involved in the 

dispute over science....................................................................................................................................... 14

5. Sample sizes, medians, modes, proportions of responses within each category of the 5-point “not 

contested” to “highly contested” scale, and Friedman’s mean rank values for each of the potentially 

contested factors in disputes over science ..................................................................................................... 18

6. Sample sizes, Friedman’s chi-square statistics, degrees of freedom, and Friedman’s mean ranks for each of 

the eight performance statements that respondents rated on a 5-point “strongly disagree” to “strongly agree” 

scale for each of the dispute resolution method categories ............................................................................ 19

7. Sample sizes and proportions of responses that had experience with dispute resolution professionals 

with each of the different affiliation types........................................................................................................ 21

8. Sample sizes, medians, modes, proportions of responses within each category of the 5-point “not useful” 

to “very useful” scale, and Friedman’s mean rank values for each of the different potential resources 

intended to help Reclamation employees resolve disputes over science ....................................................... 29
1

Disputes over Science and Dispute Resolution 

Approaches—A Survey of Bureau of Reclamation 

Employees

By Nina Burkardt and Emily W. Ruell

Executive Summary 

A growing number of constituencies with different interests and values are competing for overallocated water resources in the Western United States (NRC, 2004). As a result, these users are 

increasingly challenging the decisions of the public agencies charged with managing Western water 

resources (NRC, 2004). Because they are the primary targets in litigation surrounding water 

management decisions, agency officials are motivated to pursue the best scientific information 

throughout the decision-making process so that their decisions are not later overturned (Adler and 

others, 2001). However, scientific studies pertaining to water resources in the West often provide 

uncertain or competing results or recommendations (NRC, 2004). As a result, science can become a 

source of disputes during decision-making processes. Disputes are defined here as “vocalized or 

articulated disagreements over what ought to be done” (Ozawa, 1996, p. 220). For example, a 

disagreement could focus on study design, methodology, scope, how findings should be used in a 

management decision, and so forth. 

The Bureau of Reclamation (Reclamation) is one of the primary institutions that manages and 

allocates water resources in the West (Bowersox, 2000; Pisani, 2003). Consequently, Reclamation 

decision-making processes are frequently questioned by a variety of affected constituencies (NRC, 

2004; Henry and Conrad, 2008). Disputes often erupt over the use of science in these decision-making 

processes when interests or objectives compete, which delay decisions and often continue long after 

decisions are made (Nie, 2003; Sabatier and others, 2005). Resolving these disputes over science during 

the decision-making processes should assist decision-makers in incorporating the best scientific 

judgment in natural resource management (Mazur, 1973; Ozawa and Susskind, 1985) and reduce the 

likelihood of later challenges. 

Researchers at Reclamation and the U.S. Geological Survey conducted a literature review in 

2010 of the different methods and approaches that have been used to resolve disputes over science 

during natural resource decision-making processes, particularly for water resources (Ruell and others, 

2010). Published case studies have demonstrated that attempts to use dispute resolution methods are 

often unsuccessful at fully resolving the disputes over science, which likely prevents their broad 

application (Abrams and Berry, 1977; Roberts and others, 1984; Coglianese, 1999; Ehrmann and 

Stinson, 1999; Koontz and others, 2004; Ozawa, 2005; Sabatier, 2005; Quirk, 2005; Koontz and 

Thomas, 2006; Langfeldt, 2006; Lorie and Cardwell, 2006; van de Wetering and McKinney, 2006; 

Wagenet and Pfeffer, 2007; Hanemann and Dyckman, 2009). However, there are no empirical studies 

with large samples that have quantified how often these disputes over science arise, how often specific 

dispute resolution methods are used by managers to resolve these disputes, and how these methods have 

performed under a number of different circumstances. This information is needed in order to understand 
2

the real strengths and limitations of each dispute resolution method and help public officials choose the 

most effective approaches. 

In spring 2011, we conducted a Web-based survey of Reclamation employees who were in 

scientific, technical, and managerial job categories to determine (1) which types of disputes over science 

are occurring and how common they are, (2) which approaches have been used by Reclamation to try to 

resolve these different types of disputes, (3) how useful Reclamation employees find these approaches 

at resolving these types of disputes, (4) the final outcomes of these disputes and the decision-making 

processes that were hindered by the disputes over science, and (5) the potential usefulness of several 

different types of dispute resolution resources that Reclamation could provide for employees that 

become involved in disputes over science. The survey questions can be found in appendix 1. Given that 

there may be variation across survey respondents that correspond with professional variation (Becher, 

1994; Anderson and Bingham, 1997; von Meier, 1999), we also examined whether responses varied 

among Reclamation employees with different professional training, years of experience, or supervisory 

levels whenever sample sizes allowed.

Main Findings

• We found that the majority of the survey respondents were not in supervisory positions and had 

completed their primary professional training in engineering. Thirty percent of all respondents had 

worked for Reclamation for more than 20 years. Thus, a substantial proportion of survey 

respondents had likely worked at Reclamation for the majority of their careers. However, it is also 

possible that the employees that had worked at Reclamation for longer were more willing to 

complete the survey or were more likely to have been confronted with a dispute over science than 

those that had worked at Reclamation for less time.

• Twenty-five percent of respondents indicated that they had been involved in a dispute over science 

while working at Reclamation. This means that a substantial proportion of Reclamation employees 

in scientific, technical, and managerial positions have experienced disputes over science that 

impeded decision-making processes. Respondents that were trained in the biological sciences were 

more likely than other respondents to have been involved in a dispute over science, and those trained 

in administration and technical trades were less likely involved in a dispute. Respondents in 

supervisory positions at the time of the survey were more likely to have been involved in a dispute 

over science than nonsupervisors. 

• Native species and species listed under the Endangered Species Act (ESA, 1973) were the most 

common issue types reported in employees’ experience with disputes over science (roughly one out 

of every four responses). More than half of those reporting on a dispute over science indicated that 

the following entities were involved: non-Reclamation Federal entities (74 percent of responses), 

State entities (68 percent), water districts (71 percent), local entities (52 percent), and environmental 

groups (59 percent). The factors that respondents described as being the most contested in disputes 

over science were the validity of the inferences drawn from the science and how much uncertainty 

was in the science. The factor that was the least contested in disputes over science was the 

qualifications of the scientists that produced the science. 

• More than nine out of every ten respondents who had been involved in a dispute over science 

reported experience with at least one of the dispute resolution methods categories. In one-third of the 

responses, the dispute resolution method selected was “direct discussions between scientists.” The 

next most common dispute resolution method type was “public education, data sharing, and results 

dissemination.” The dispute resolution method types that were the least common were “adaptive 

management” and “independent expert review.” 
3

• “Active collaboration,” “public education, data sharing, and results dissemination,” and “adaptive 

management” generally involved a greater number of party types than the other method types, but 

did not involve a greater number of issue types. There were usually fewer party types involved if 

there was no method used to resolve the dispute than if a dispute resolution method had been used. 

• All of the dispute resolution method types appeared to have similar strengths and weaknesses in 

resolving disputes over science. All method types were rated highest in how they were able to foster 

communication between parties about the science under dispute. All method types were also rated 

relatively high in their ability to increase the credibility of Reclamation’s position on the science 

under dispute. Out of the eight different performance criteria, all method types were rated the lowest 

on their ability to resolve differences over the interpretation of the science, even though this factor 

was reported to be one of the most contested factors in disputes over science. In addition, all method 

types were also rated relatively low on how well they incentivized the parties to reach an agreement 

on the science under dispute. Despite this, all method types were rated somewhat higher for their 

ability to allow the process to move forward to a decision, regardless of whether agreement was 

reached about the science. 

• The overall evaluation for each of the dispute resolution method types was neutral to somewhat 

positive, although there was a lot of room for improvement with every method type. Supervisors 

generally had more positive evaluations of the use of dispute resolution methods than 

nonsupervisors. 

• The use of dispute resolution professionals appeared to be somewhat infrequent (reported by less 

than 17 percent of respondents). When they were used, respondents usually rated dispute resolution 

professionals as having a somewhat positive effect on the dispute over science. 

• The majority of the disputes over science and the decision processes hindered by these disputes over 

science were still ongoing. The disputes that were resolved were most commonly resolved within 

Reclamation and at the level at which they originated. Respondents that had worked at Reclamation 

for less than five years were more likely than expected to indicate that the decision-making process 

was ongoing. Respondents that had worked at Reclamation for more than 10 years were more likely 

than expected to indicate that the decision had been made. 

• There was a clear association between the outcome of the dispute over science and the outcome of 

the decision hindered by the dispute over science. In the majority of respondents’ experiences, when 

the dispute was resolved, the decision was also made. The disputes that were still ongoing were 

almost always associated with decision processes that were still ongoing. Finally, when the dispute 

was not resolved and unlikely to be resolved, the decision was usually not made either. 

• There was no indication that any of the dispute resolution method types resulted in substantially

different outcomes in disputes over science than other types of disputes. Furthermore, there was also 

no apparent difference in the outcomes of disputes for which a dispute resolution method was 

attempted versus disputes for which no method was used. Similarly, there was no apparent 

difference between dispute resolution methods on the outcomes of the decisions that had been 

hindered by the dispute over science, and there was no apparent difference in the outcomes of 

decision processes when a dispute resolution method had been used versus those for which no 

method had been used. However, please note that the selection of the dispute resolution method 

types across disputes was not random; therefore, disputes for which no method type was used may 

have differed from the other disputes. 

• Half of the respondents had experience with multiple dispute resolution method types in either the 

same or different disputes over science. Biological and physical scientists were more likely than 
4

respondents with other primary professional training backgrounds to have experience with multiple 

dispute resolution method types. 

• Respondents rated a training class on resolving disputes over science and a bibliography on this 

subject as the most useful resources out of the potential resources that were suggested in the survey. 

Although these survey results may be used to help predict which future decision-making 

processes are most likely to involve a dispute over science, which Reclamation employees are most 

likely to experience a dispute over science, and which aspects of the science under dispute are most 

likely going to be contested, they did not indicate that any dispute resolution method was more effective 

at resolving disputes over science than any other. Future research is needed to determine whether there 

are additional variables underlying these disputes that were not measured in this survey that may 

identify when dispute resolution methods are most effective, or whether resolving aspects of these 

disputes, such as differing interpretations of science, is very difficult or impossible regardless of the 

dispute resolution methods used. 

Background 

Characteristics of Disputes over Science

At the outset, resolving disputes over science usually requires identification of (1) the types of 

issues informed by the science that is under dispute, (2) the set of parties that are involved in the 

dispute, and (3) what aspects of the science are being disputed (Ruell and others, 2010). Ruell and 

others (2010) concluded that the most effective dispute resolution method likely depends on the type of 

dispute over science that arises. The dispute resolution methods that work well in some disputes may 

not work well in others because of these and other factors. 

Issues Involved

Disputes over science can arise because of a number of different underlying issues. For example, 

when the use of a resource by one party automatically limits the use of the resource by another (that is,

zero-sum resources), enduring disagreements often arise between competing parties over the use of 

science in decision-making for these resources (Quirk, 2005; Hanemann and Dyckman, 2009). In 

contrast, disputes over science may be less likely when resources are plentiful and may be relatively 

easy to resolve when they do arise. Disputes can also involve large, complex social and ecological 

systems where scientific uncertainty is unavoidably high (Johnson, 1999). These complex systems 

usually do not have straightforward technological solutions for improving management decisions 

(Johnson, 1999; Schmidtz, 2000; Nie, 2003). Furthermore, increasing recognition of the importance and 

complexity of ecological and social interactions has led to even greater uncertainty (Johnson, 1999; Ison 

and others, 2007). Yet, disputes may still arise in systems that are well studied and understood due to 

competing interests. However, the approaches required to resolve disputes involving well-studied 

systems may be different than systems where the science contains much uncertainty, regardless of the 

underlying cause of the dispute. 

Parties Involved

How to effectively resolve disputes over the use of science may depend on the number and types 

of competing parties involved in the dispute. Types of parties can be defined by their differing beliefs, 

ideologies, groups, organizations, disciplines, communities, scientific training, or cultures. Some parties 

represent larger organizations or the “public interest,” whereas others only represent themselves (van de 
5

Wetering and McKinney, 2006). Some parties are paid to participate; others must invest their own time 

and resources (van de Wetering and McKinney, 2006). Some representatives are willing and able to be 

flexible and modify their preferred decision alternatives, whereas others are unwilling or unable to be 

flexible because they have fixed commitments to their members, leaving no room for compromise 

(Roberts and others, 1984). There may also be subgroups within organizations that have different 

perspectives and come to different conclusions from the same science (von Meier, 1999). 

Larger numbers of parties involved in a dispute can greatly increase the difficulty of organizing 

negotiations, particularly with ad hoc processes that do not have stable legal and administrative 

precedents (Quirk, 2005). Also, the parties involved in a dispute may change over time, which can 

greatly complicate attempts to incorporate revolving, frequently nonrepresentative groups of 

stakeholders in decision-making processes, particularly for time-consuming dispute resolution processes 

(Burkardt and others, 1995; Koontz and others, 2004; Quirk, 2005; Koehler and Koontz, 2008). 

The Science under Dispute

There are many aspects of science itself that may cause a dispute among parties involved in a 

decision-making process. The following list is by no means exhaustive, but we argue that it touches on 

many of the likely issue areas behind disputes over science. First, parties may disagree on whether the 

existing science addressed all of the critical issues involved in the decision-making process. For 

example, the parties may disagree on whether or not the scientific studies measured all of the important 

variables. Second, parties may disagree on the appropriate quality of the data used in the decisionmaking process. The best science available for decision-making processes may range from anecdotal 

information to observational studies to peer-reviewed scientific experiments (Ruckelshaus and others, 

2002; Ozawa, 2005; Henry and Conrad, 2008). Third, parties may disagree on the interpretations of the 

best available science as either too limited or too far-reaching (Quirk, 2005). Fourth, and along the same 

lines, parties may disagree on how much uncertainty there is in the science (Johnson 1999). Fifth, 

parties may disagree on whether more scientific studies are necessary before decision-making should 

take place. Sixth, parties may disagree on whether science is an appropriate basis for decision-making or 

whether decisions should be based on other factors. Although some parties may prefer that science is 

not required to justify an action, others may prefer to make decisions based primarily on the science 

available (Roeder, 2005; Stiftel and Scholz, 2005). Finally, parties may disagree on whether the 

scientists that produced the science were qualified, and by extension, whether the science itself is 

credible (Quirk, 2005). 

Dispute Resolution Methods

Numerous methods for resolving disputes during decision-making processes have been proposed 

in the last several decades, and many of them have also been ground tested. Because some of these 

methods have been given multiple names by different academic or professional disciplines, overlap one 

another in methodology, or are simply too nebulous to be applied uniformly, we have developed the 

following broad categories or types of dispute resolution methods in table 1 from the literature on 

dispute resolution approaches (Ruell and others, 2010). These categories are differentiated by what 

activities and what parties the methods involve (table 1). 
6

Table 1. Categories of dispute resolution methods defined by what parties and what activities they involve.

Dispute resolution method category Definition

Direct discussions between or among 

scientists

Processes that bring technical experts involved in a science dispute together to 

identify areas of agreement and disagreement, data needs and gaps, scientific 

protocols, and potential approaches to resolving technical disputes.

Independent expert review One or more outside experts review the disputed science and reach conclusions 

regarding the weight of the evidence and the adequacy of the science.

Reclamation undertakes more science 

and analysis independently of the 

other parties

Reclamation undertakes additional studies or analyses in an effort to address 

concerns or conflicts.

Active collaboration in research and 

analysis

Outside parties involved in the dispute engage with Reclamation in collaborative 

science, jointly undertaking scientific training, hypothesis development, data 

collection, model building, or data analysis.

Public education, data sharing, and 

results dissemination

Any outreach activity designed to inform the public and stakeholders about the 

technical issues, existing data and science, and Reclamation's analysis of the 

information.

Adaptive management

Uncertainties or differences over science in the management of water and related 

resources are addressed using planned programs of experimentation and adaptive 

decision-making based on scientific feedback.

Defining Dispute Resolution

Just as not all disputes over science are the same, dispute resolution processes may vary widely. 

Scholars have argued that there are many aspects to a dispute or the relationships between parties in a 

dispute that may improve even if the parties do not come to complete agreement (Roberts and others, 

1984). Thus, scholars agree that a suite of effects should be used to evaluate successes and failures of 

different dispute resolution methods (Roberts and others, 1984; Ury and others, 1988; van de Wetering 

and McKinney, 2006). Most published case studies of dispute resolution methods have shown that these 

methods have mixed performance in resolving disputes (Abrams and Berry, 1977; Roberts and others, 

1984; Coglianese, 1999; Ehrmann and Stinson, 1999; Koontz and others, 2004; Ozawa, 2005; Sabatier 

and others, 2005; Quirk, 2005; Koontz and Thomas, 2006; Langfeldt, 2006; Lorie and Cardwell, 2006; 

van de Wetering and McKinney, 2006; Wagenet and Pfeffer, 2007; Hanemann and Dyckman, 2009). 

For example, the use of a dispute resolution method may improve communications between parties 

about the science under dispute but may fail at getting parties to reach a common interpretation of that 

science.

The performance criteria evaluated in this study were identified from the literature on dispute 

resolution processes (Ruell and others, 2010). These criteria ranged from whether the resolution method 

improved relationships among parties in the dispute, regardless of whether the dispute was ultimately 

resolved, to the logistical feasibility of the method, to whether the method allowed the decision-making 

process that was delayed by the dispute to proceed to a decision. How well each method performs for 

each of these performance criteria may be more important to Reclamation in some situations than in 

others. We used respondents’ evaluations of eight different performance measures to get a sense for how 

these resolution methods have performed relative to other resolution methods in different types of 

disputes over science. 
7

Web Survey 

We conducted a Web survey of current Reclamation employees that could potentially have been 

involved in a dispute over science during a decision-making process that involved parties outside of 

Reclamation. We used the program Key SurveyTM version 7.1 (WorldAPP, Braintree, Mass.) to 

administer the survey. Survey recipients were emailed a unique link (that is, a URL) that took them to a 

Web page that administered their survey. Recipients were allowed to leave and return to their saved 

survey at later dates until they submitted their survey. 

Given that this online survey targeted Federal employees with advanced educational 

backgrounds and job-provided email and Internet access, we expected our response rate to be relatively 

high and comparable to response rates found for other professional and university populations (Yun and 

Trumbo, 2000; Shih and Fan, 2008). We did not conduct a nonresponse survey, because we expected 

that the representativeness of our sample would have minimal bias due to the relatively similar socioeconomic or educational status of the survey population of professionals (Yun and Trumbo, 2000). 

The Web survey was structured to ensure that respondents would only answer questions about 

disputes over science in which they had some direct involvement, rather than disputes that they knew 

about secondhand. This was done to try to improve the accuracy of respondents’ recounting of events 

and outcomes. Furthermore, we limited respondents to describing disputes over science that included 

both outside agencies or stakeholders and that were serious enough to impede a water-resource 

management decision, as these were the disputes in which Reclamation had the most interest in learning 

more about. The survey was divided into units composed of questions addressing different broad topics. 

Respondents were navigated through survey question units based on their answers to certain questions 

within each unit. The majority of the questions were closed-ended questions, which are the preferred 

format in surveys because they make responses comparable, and thus, easier to analyze using statistical 

analysis methods (Dillman and others, 2009). However, because the subjects of the survey were 

relatively complex, we included some open-ended questions that allowed the respondents to provide 

additional information on each part of the survey and on disputes over science in general. Whereas 

open-ended questions can increase the rate of incomplete surveys and can be difficult to analyze 

(Dillman and others, 2009), online surveys generally stimulate longer and more detailed comments than 

traditional mail-based surveys (Yun and Trumbo, 2000). With the exception of the question that asked 

respondents to describe the scientific issues that were under dispute, we did not analyze the responses to 

open-ended questions in this report. 

Topics of Survey Question Units

Unit 1

The purpose of the first question unit was to determine whether or not respondents had ever been 

directly involved in a dispute over science, and if yes, then whether a dispute resolution method was 

used to try to resolve at least one of these disputes. 

Unit 2 (Beginning of Loop 1)

This question was used to determine the categories of dispute resolution methods with which 

respondents have the most experience. 
8

Unit 3 (Beginning of Loops 2 and 3)

Respondents were asked to describe the nature of the most recent dispute over science in which 

they had been directly involved and that used a method within the dispute resolution method category 

that they had selected in unit 2. 

Unit 4

This question unit asked respondents to evaluate how well they thought the dispute resolution 

method performed in the dispute over science. 

Unit 5

This unit of questions asked respondents if a dispute resolution professional (DRP) had been 

used in this dispute, and if yes, the effect of the DRP had on the dispute. 

Unit 6

This unit of questions asked respondents to indicate the final outcome of the dispute over 

science. If the dispute was resolved, then respondents were asked to identify where the dispute was 

resolved and whether it escalated to a higher level of authority. 

Unit 7

This unit of questions asked respondents to indicate the final outcome of the water-management 

decision that was hindered by the dispute and to provide any additional comments about this particular 

dispute over science. 

Unit 8

This unit of questions asked respondents whether or not they had been involved in a dispute that 

had used one of the other dispute resolution methods from another category. 

Unit 9

This unit of questions asked respondents to comment on disputes over science in general and to 

evaluate how useful they thought several potential dispute resolution resources would be for 

Reclamation employees.

Unit 10

The final unit contained questions on respondents’ professional background and experience. 

Respondents were then directed to submit their survey.

Survey Launch

We selected 2,799 Reclamation employee email addresses to receive a survey invitation based 

on the job title categories they held on November 28, 2010, the date the email distribution list was 

compiled. These categories included all administrators, managers, specialists, technicians, scientists, 

economists, and engineers. Our selection of job title categories likely included many Reclamation 

employees who have not been involved in disputes over science, but we did not want to risk excluding a 

group that may have a different experience with these disputes over science than the others. The survey 

invitation email specifically asked that employees enter the survey and answer a few questions about 

their professional background even if they had not been involved with a dispute over science. Positions 
9

that clearly solely involved office administrative tasks were excluded because it was unlikely that these 

employees would be involved in these types of disputes. 

On February 17, 2011, five days prior to the launch of the survey, a pre-notice email was sent by 

the Director of Reclamation’s Research and Development Office to all survey recipients. This email 

encouraged employees to participate in the survey. On February 22, 2011, the invitation email was sent 

to all recipients with a link to the survey, which forwarded respondents to a Web site where the survey 

was hosted. Email links were user-specific so that the respondent could re-enter, view, and modify 

answers until the survey was finally submitted. Multiple submissions from the same link were 

prohibited. Follow-up reminder emails that included the original link were sent at approximately one 

(February 28, 2011), two (March 7, 2011), three (March 15, 2011), four (March 22, 2011), and five 

weeks (March 28, 2011) after the launch of the survey to the recipients that had not yet started or 

completed their survey. Previous studies have shown that the day of the week that the survey 

correspondence is received does not affect response rates, response time, or response quality (Shinn and 

others, 2007), and that the majority of responses are received within the first few days after invitation 

emails are sent (Yun and Trumbo, 2000; Shih and Fan, 2008). Nevertheless, we avoided sending survey 

emails on holidays or weekends so that the survey correspondence was less likely to be received during 

employees’ vacations or buried within a pile of other emails received while employees were out of the 

office. We did not include any gift incentives for recipients, because Federal employees are prohibited 

from accepting them, and they do not appear to significantly increase response rates for electronic 

surveys (Yun and Trumbo, 2000; Baruch and Holtom, 2008). 

Study Objectives and Results

There were 1,645 respondents that completed their surveys and 100 that started but did not 

complete their surveys. Of those respondents that did not start their survey, there were 1,044 that were 

considered implicit refusals and 10 that were verified as noncontacts for the duration of the survey based 

on the content of their automated email replies. Thus, our calculated minimum response rate was 59 

percent. This response rate was high relative to other electronic surveys that have been conducted (Yun 

and Trumbo, 2000) and was higher than the average response rates for both Web surveys (34 percent) 

and mail surveys (45 percent) in the meta-analysis conducted by Shih and Fan (2008). 

Respondents’ Professional Background and Experience

Objectives and Hypotheses

The objective was to determine the professional background and experience of respondents, 

including primary professional training, supervisory experience, and the number of years respondents 

had worked for Reclamation. We did not have any predictions regarding the professional background 

and experience of respondents.

Summary of Findings

• Engineering was the most common primary professional training of respondents, followed by 

administration (including contracts and realty), and biological sciences. 

• The majority of respondents were not in supervisory roles at the time of the survey. 
10

• Approximately one-third of respondents had been employed for five years or less at Reclamation, 

but an almost equal number had worked there more than 20 years. However, the second-most 

common category of years working at Reclamation selected by respondents was over 20 years. 

Detailed Results

The number and percentage of respondents within each category of the professional background 

and experience variables are presented in table 2. Some of the categories for the primary professional 

training were not in the original survey but were coded from “other, please specify” responses or 

combined (see appendix 2). Please note that these statistics do not represent all Reclamation employees 

at the time of the survey but only the Reclamation employees within the job categories that were 

selected to receive the survey.

Table 2. Sample sizes and proportions of responses within each of the professional background and experience

variables categories. 

Professional background and demographic variables and categories Number of 

responses

Percent1

Supervisory position (n = 1,633)

No 1,304 80

Yes 329 20

Primary professional training (n = 1,637)

Engineering 714 44

Administration/Business/Contracts/Realty 358 22

Biological sciences 199 12

Physical sciences 149 9

Social sciences 113 7 

Technical trades/Visitor services 60 4 

Law 22 1 

Other 10 1 

Architecture 6 0 

Computer science 4 0 

Law enforcement 2 0 

Years worked at Reclamation at time of survey (n = 1,637)

0–5 years 524 32

 6–10 years 302 18

11–15 years 191 12

16–20 years 125 8 

>20 years 495 30

1

Percentage within each category within each variable. 

A generalized linear model (GZLM) for respondents’ number of years worked at Reclamation 

(0–5 years, 6–10 years, 11–15 years, 16–20 years, >20 years) revealed that the predicted main effect of 

primary professional training was significant (Wald χ

2

(10, n = 1,627) = 32.85, p <0.001), and the 

predicted main effect of supervisors was significant (Wald χ

2

(1, n = 1627) = 21.63, p <0.001). It is 

possible that there were gender differences in the professional training and supervisory experience 
11

variables that may have confounded the associations found in the analysis above and elsewhere in this 

report, but we were not able to ask respondents to indicate their gender in this survey.

Involvement in Disputes over Science

Objectives and Hypotheses

Our main objectives were to assess (1) how common it has been for Reclamation employees in 

scientific and technical positions to have direct experience with one or more disputes over science and 

(2) whether certain types of employees in scientific and technical positions in Reclamation have been 

more likely to be involved in disputes over science than others. 

Specifically, we predicted that the respondents trained in “biological sciences” would have been 

more likely to be involved in disputes than others because they would be involved in conducting 

scientific studies for, or implementing, environmental laws, such as the Endangered Species Act (ESA, 

1973) and the National Environmental Policy Act (NEPA, 1969). These laws explicitly require that 

decision-making processes utilize the best available science and involve other agencies and the 

interested public. In contrast, we expected that respondents trained in “administration, business, realty, 

and contracts” and in “technical trades and visitor services” would be less likely to have been involved 

in disputes over science than the other professional training categories because they would be less likely 

to be involved in implementing laws that require science to be the basis of decisions. 

We also predicted that the respondents in supervisory positions would be more likely to have 

been involved in a dispute over science at some point than nonsupervisors because supervisors likely 

have an increased exposure to disputes via their supervisees. 

Finally, we predicted that the respondents that had worked for Reclamation for a greater period 

of time would have been exposed to more situations or decisions that would potentially result in a 

dispute over science. 

Summary of Findings

• Roughly one out of every four respondents had been involved in a dispute over science. Again, this 

does not represent the proportion of all Reclamation employees that have experienced a dispute over 

science but rather the proportion of Reclamation employees within the job categories that were 

selected to receive the survey.

• Respondents with primary professional training in “biological sciences” were more likely to have 

been involved in a dispute than respondents with other types of primary professional training. 

Respondents with professional training in “administration/business/realty/contracts” and “technical 

trades/visitor services” were less likely to have been involved in a dispute than respondents with 

other types of professional training. 

• Respondents in supervisory positions were more likely to have been involved in a dispute over 

science than nonsupervisors. 

• We found that the more years that a respondent had worked for Reclamation, the higher their 

probability of having been involved in a dispute over science. 

Detailed Results

Respondents to all complete and incomplete surveys (n = 1,745) answered the first mandatory 

survey question, which asked whether or not respondents had ever been involved in a dispute over 
12

science. From these surveys, 25 percent of the respondents (n = 438) indicated that they had been 

involved in a dispute over science that included parties outside of Reclamation. 

The GZLM revealed that the predicted main effect of primary professional training was 

significant (Wald χ

2

(10, n = 1,627) = 99.63, p <0.001). Post hoc z tests of the contingency table of 

involvement in a dispute over science by primary professional training (Pearson’s χ

2

(10, n = 1,637) = 

128.04, p <0.001) revealed that respondents with professional training in “biological sciences” were 

more likely to have been involved in a dispute than respondents with other types of professional 

training, and respondents with professional training in “administration/business/realty/contracts” and 

“technical trades/visitor services” were less likely to have been involved in a dispute than respondents 

with other types of professional training. The GZLM also revealed that the predicted main effect of 

supervisors was significant when taking into account primary professional training (Wald χ

2

(1, n = 

1,627) = 28.87, p <0.001). Of the 329 respondents that were supervisors, 29 percent (n = 95) indicated 

that they had been involved in a dispute over science, and only 17 percent (n = 215) of the 1,304 

nonsupervisors had been involved in a dispute over science. Finally, the GZLM revealed that the 

predicted main effect of the number of years respondents had worked at Reclamation was significant 

when taking into account both primary professional training and supervisory position (Wald χ

2

(4, n = 

1,627) = 18.35, p = 0.001). 

Dispute Characteristics

Objectives and Hypotheses

The main objective was to measure the scope of the disputes over science in which Reclamation 

employees have been involved, including both the number and types of issues that commonly underlie 

disputes over science, and the different types of parties outside of Reclamation that were usually 

involved in the disputes. 

We predicted that native and endangered species would be the most common issue types 

underlying disputes over science because the requirements of the ESA (1973) and NEPA (1969) often 

compete with interests of other parties.

We predicted that other Federal agencies, water districts, and State agencies would be the 

outside parties most frequently involved in respondents’ disputes due to the requirements of the ESA 

(1973) and NEPA (1969). 

Summary of Findings

• The large majority of respondents (81 percent) described only one type of issue. Fewer respondents 

(16 percent) reported two issue types, and fewer still reported three or four issue types (<2 percent).

• The most frequent types of issues reported in the responses included native species (25 percent; 

including all aquatic and terrestrial plant and animal species), endangered or threatened species 

protected under the ESA (1973) (23 percent), surface-water measurements (15 percent), and natural 

hazard risks (11 percent). The least frequent types of issues reported by respondents in less than 1 

percent of the responses were Native American rights, economic analyses, climate change effects, 

aquaculture, and dam removal. 

• The disputes over science described by respondents usually involved multiple types of parties 

outside of Reclamation. In the majority of responses, Federal entities, water districts, State entities, 

environmental groups, and local entities were involved in the disputes over science. 
13

Detailed Results

Out of the 100 incomplete surveys, 29 respondents had at least completed questions from units 

1–7 (they had completed loop 1 of the survey) for a dispute over science. Therefore, these responses 

were considered complete enough and were included with the completed surveys in further analyses on 

the characteristics of the disputes and respondents’ evaluation of the different dispute resolution 

methods. 

Out of the 347 respondents that had experience with disputes over science, 83 percent (n = 288) 

reported a range of one to four issue types. The large majority, 81 percent of respondents (n = 234), 

described only one issue type. However, many respondents reported that multiple issue types were 

involved in the dispute: 16 percent (n = 46) of respondents reported two issue types, 1 percent (n = 4) of 

respondents reported three issue types, and 1 percent (n = 4) of respondents reported four issue types. 

The number and percentage of respondents that reported each issue type is presented in table 3.

Table 3. Sample sizes and proportion of responses that indicated that each scientific issue type had been involved 

in the dispute over science (check all that apply).

Issue type(s) underlying disputes over science1 Number of 

responses

Percent2

Native species 72 25

Endangered species 65 23 

Surface-water aspects (storage, flow, quality, and so forth) 44 15

Natural hazard risk (seismic activity, drought, flooding, and so forth) 32 11

Geomorphology 25 9 

Agriculture 19 7 

Water rights 16 6 

Non-native species 14 5 

Materials engineering (quality, durability, and so forth of materials used) 14 5 

Human health risk due to toxicity or disease 12 4 

Groundwater aspects (levels, quality, connections, and so forth) 11 4 

Cultural resources 6 2 

Hydropower 4 1 

Recreation 4 1 

Desalinization plants 4 1 

Economic analyses 3 1 

Climate change effects 3 1 

Native American rights 2 1 

Aquaculture 2 1 

Dam removal 2 1 

1

Responses to an open-ended question on the issues underlying the dispute were coded into general categories of issues by 

authors NB and ER. Many disputes involved more than one issue type (range of one to four issue types). 2

The percentage of responses (n = 288) within each issue type. 
14

Multiple types of parties outside of Reclamation were usually involved in disputes over science 

(M = 4.73, SD = 2.48, range = 1–11 party types). The number and percentage of respondents that 

reported each party type is presented in table 4.

Table 4. Sample sizes and proportions of responses that indicated that each type of party was involved in the 

dispute over science (check all that apply).

Outside party type(s) involved in disputes over science1 Number of 

responses

Percent2

Federal entity 254 74

Water district 241 71

State entity 231 68

Environmental interest group 202 59

Local entity 178 52

Tribal entity 111 33

Business entity 100 29

Individual citizens 92 27

Academia 90 26

Recreation interest group 84 25

International entity 27 8 

Elected officials 7 2 

1

Respondents were instructed to select all non-Reclamation party types that were involved in the dispute. Most disputes 

included multiple party types (range 1–11 party types). 2

The percentage of responses (n = 342) within each party type.

Dispute Resolution Methods

Objectives and Hypotheses

The primary objectives were to determine (1) whether the use of dispute resolution methods was 

common in disputes over science involving Reclamation employees and (2) which of the methods types 

were the most common in Reclamation employees experiences. We did not have any a priori

predictions regarding the use of different dispute resolution method types in Reclamation.

Summary of Findings

• The majority of respondents with direct experience with disputes over science indicated that some 

sort of dispute resolution method had been used to try to resolve the dispute over science. 

• The most common dispute resolution method categories selected by respondents were “direct 

discussion between scientists” and “public education, data sharing, and results dissemination.” 

Detailed Results

From the group of respondents that had been involved in a dispute over science and had 

completed question units 1–7 (n = 347), 324 (93 percent) had experience with the use of at least one of 

the dispute resolution methods from the categories listed, and 23 (7 percent) had no experience with any 
15

of the dispute resolution method categories. None of the respondents indicated that they had used a 

method that did not fall into one of the six categories provided. The dispute resolution method category 

that respondents had the most experience with was “direct discussion between scientists” followed by 

“public education, data sharing, and results dissemination” (fig. 1). The method categories that 

respondents had the least experience with were “adaptive management” and “independent expert 

review.” 

Figure 1. Proportion of respondents that selected each of the dispute resolution method types. 

Number of Issue and Party Types Involved in Dispute Versus Method Types Used

Objectives and Hypotheses

The primary objectives were to determine whether certain dispute resolution method types were 

more likely to involve (1) fewer or more issue types or (2) fewer or more party types than the others.

We predicted that dispute resolution method types that generally require agencies to include 

stakeholders and interest groups in their implementation, such as “active collaboration,” “public 

education, data sharing, and results dissemination,” and “adaptive management” (Ruell and others, 

2010), were more likely to be associated with disputes that involved more issue types and party types 

outside of Reclamation than method types that did not include outside parties, such as “direct discussion 

between scientists” and “Reclamation undertakes more science independently of the other parties.” 

We predicted that disputes over science for which no dispute resolution method was used would 

have the smallest mean number of outside party types involved because disputes that involved larger 

numbers of outside parties would be more likely to have one or more parties call for or initiate some sort 

of dispute resolution process. 

EXPLANATION
16

Summary of Findings

• There was no apparent association between the number of issue types involved and the dispute 

resolution method type that was used to resolve the disputes over science. 

• In general, dispute resolution method types that usually require agencies to include stakeholders and 

interest groups in the dispute resolution process had a larger mean number of party types outside of 

Reclamation involved. In particular, the method type “adaptive management” was associated with 

the largest mean number of outside party types.

• Disputes over science for which no method was used to resolve the dispute had a smaller mean 

number of party types outside of Reclamation involved in the dispute than those for which a dispute 

resolution method had been attempted. 

Detailed Results

The numbers of issue types that were involved in respondents’ experiences with disputes over 

science did not significantly differ between the dispute resolution method types, contrary to our 

expectation (Pearson’s χ

2

(18, n = 286) = 13.47, p = 0.763). 

The mean number of party types that were involved in the disputes did vary significantly among

the different types of dispute resolution methods used (F(6, 335) = 3.832, p = 0.001). The mean number 

of party types involved for “adaptive management” was significantly higher than the mean for disputes 

with no dispute resolution method, “direct discussion between scientists,” and “Reclamation undertakes 

more science independently of the other parties” in Tukey post-hoc tests with p < 0.05 (fig. 2). 

Furthermore, respondents that indicated that no method was used to resolve the dispute tended to be 

involved in disputes with a significantly smaller mean number of parties involved (M = 3.57, 

SD = 1.83) than respondents with experience with a dispute resolution method (M = 4.80, SD = 2.50; 

t(340) = –2.06, p = 0.040). 

Figure 2. Means and 95-percent confidence intervals (CI) for the number of non-Reclamation party types involved 

in disputes over science within each of the dispute resolution method categories.
17

Contested Factors in Dispute over Science

Objectives and Hypotheses

The main objectives were (1) to determine what aspects or characteristics of science were most 

contested between parties in the disputes over science and (2) whether the reported levels of 

contestation of any of the factors in the dispute differed across the dispute resolution method types. We 

did not have any a priori predictions on the relative contestation of different characteristics of the 

disputes over science experienced by Reclamation employees.

Summary of Findings

• The four factors that respondents rated the highest on the 5-point “not contested” to “highly 

contested” scale were “the inference(s) drawn from the science,” followed by “the level of 

uncertainty in the science,” “whether or not the existing science addressed the critical issues,” and 

“the quality of the data used.” More than 50 percent of respondents rated these four factors as either 

a value of 4 or 5 (contested to highly contested). The factor that respondents rated the lowest on the 

scale was “the qualifications of the scientists who produced the science.” 

• The dispute resolution method type used did not appear to be associated with the degree to which 

any of the seven factors were contested.

Detailed Results

There was a significant difference in how respondents rated each of the seven potentially 

contested factors when describing their disputes over science (Friedman’s χ

2

(6, n = 264) = 262.44, p

<0.001 excluding responses that selected “I don’t know” or “not applicable” for any of the seven 

factors; table 5). A small percentage of the respondents gave “not applicable” or “I don’t know” answers 

for each of the factors (range 2–5 percent of respondents). 

The distribution of respondents’ ratings did not significantly differ among the dispute resolution 

method categories for any of the seven factors (the Kruskall-Wallis one-way ANOVA (analysis of 

variance) tests were not significant with p ≥0.161; table 5).

In addition to the seven potentially contested factors listed in the survey, respondents were given 

the option of adding an “other, please specify” response. These additional options were not evaluated by 

respondents on the “not contested” to “highly contested” scale and are not included in table 5 or in the 

analysis. 

Evaluations of Performance of Dispute Resolution Method Types

Objectives and Hypotheses

The main objectives were to determine (1) which aspects of the disputes over science each of the 

dispute resolution methods were the most or the least successful at resolving, (2) whether any of the 

dispute resolution method categories were better or worse at resolving disputes over science than the 

others, (3) whether differences in respondents’ experiences were associated with their professional 

background and experience. We did not have any a priori predictions for these three objectives. 
18

Table 5. Sample sizes, medians, modes, proportions of responses within each category of the 5-point “not 

contested” to “highly contested” scale, and Friedman’s mean rank values for each of the potentially contested 

factors in disputes over science.

Potentially contested factors in 

disputes over science

Number of 

responses1 Median Mode

Scale

“not contested” to “highly 

contested” Freidman’s 

mean rank2

1 2 3 4 5

1. Whether or not the existing 

science addressed the critical 

issues

315 4 4 11% 12% 18% 33% 26% 4.40

2. The quality of the data used 323 4 4 11% 11% 26% 33% 20% 4.18

3. The inference(s) drawn from 

the science 321 4 4 5% 7% 17% 39% 32% 5.07

4. The level of uncertainty in 

the science 322 4 4 7% 10% 25% 36% 22% 4.41

5. The need for additional 

scientific investigation 324 3 4 21% 14% 19% 30% 17% 3.76

6. Whether or not science 

should be the basis for the 

management decision

315 3 1 28% 16% 18% 17% 21% 3.47

7. The qualifications of the 

scientists who produced the 

science

311 2 1 34% 24% 19% 14% 9% 2.70

1

The number of responses for each factor and the calculated medians, modes, and percentages within each category in the 5-

point not contested to highly contested scale do not include the "not applicable" and "I don't know" responses for that factor. 

2

Calculated from the responses (n = 264) that did not select "I don't know" or "not applicable" for any of the seven factors.

Summary of Findings

• There were significant differences in respondents’ ratings across the eight different performance 

statements for all of the method types. Furthermore, ratings for each of the eight statements 

remained fairly consistent across all of the method types. Across method types, the resolution 

statements to which respondents gave the highest agreement scores for all of the method types were 

“the method fostered communication between the parties about the scientific issues,” and “the 

method increased the scientific credibility of Reclamation’s position.” The resolution statements to 

which respondents gave the lowest agreement scores for almost all of the method types were “the 

method resolved differences over the interpretation of the science,” and “the method created 

incentives to reach agreement on the dispute over science.” 

• There was no dispute resolution method category that appeared to resolve disputes substantially 

better or worse than any others, including the category “no method.” However, please note that the 

selection of the dispute resolution method types across disputes was not random; therefore, the 

disputes may have differed in some way across groups. 

• For the most part, the evaluations of dispute resolution method types were slightly positive. 

• Respondents’ evaluations of the dispute resolution methods did not vary by primary professional 

training, supervisory position, or the number of years that respondents had worked at Reclamation. 
19

Detailed Results

The results of the analyses of respondents’ ratings for each of the performance statements are 

presented in table 6. 

Table 6. Sample sizes, Friedman’s chi-square statistics, degrees of freedom (d.f.), and Friedman’s mean ranks for 

each of the eight performance statements (that is, Likert items) that respondents rated on a 5-point “strongly 

disagree” to “strongly agree” scale for each of the dispute resolution method categories. 

Dispute resolution 

method type used

Number of 

responses1

Friedman's 

chi-square

d.f. p-value

Friedman’s mean rank for performance statements 1–82

1 2 3 4 5 6 7 8

Direct discussion 

between or among 

scientists

113 121.77 7 <0.001 6.27 3.52 3.78 4.97 4.27 4.67 4.22 4.29

Independent expert 

review 33 36.86 7 <0.001 5.59 3.89 3.29 5.65 4.17 4.24 4.35 4.82

Reclamation 

undertakes more 

science and 

analysis 

independently of 

the other parties

42 46.00 7 <0.001 5.75 3.71 3.90 5.69 3.58 3.98 4.58 4.80

Active 

collaboration in 

research and 

analysis

37 41.20 7 <0.001 6.38 3.81 3.74 4.89 3.84 4.07 4.55 4.72

Public education, 

data sharing, and 

results

dissemination

47 30.45 7 <0.001 5.64 3.46 3.96 4.72 4.68 4.48 4.69 4.37

Adaptive 

management 34 32.34 7 <0.001 5.82 3.65 3.53 4.97 4.10 4.07 4.96 4.90

1

Does not include responses that selected "I don't know" or "not applicable" for any of the eight resolution statements.

2

Statement 1: The method fostered communication about scientific issues.

 Statement 2: The method resolved differences over interpretation of the scientific results.

 Statement 3: The method created incentives to reach agreement on the dispute over science.

 Statement 4: The resolution method increased the scientific credibility of Reclamation's position.

 Statement 5: The time required for this method was reasonable for Reclamation.

 Statement 6: The money required for this method was reasonable for Reclamation.

 Statement 7: The method made it possible to move forward in the absence of agreement over the science.

 Statement 8: The method helped the process proceed to the decision delayed by the dispute over science.
20

The mean value for the summated performance scale was significantly greater than zero (the 95-

percent confidence intervals did not overlap zero) for all of the method categories except “Reclamation 

undertakes more science independently of the other parties” and “adaptive management” (fig. 3). The 

means for the summated performance scale did not significantly differ across the dispute resolution 

method categories (F(5, 300) = 0.381, p = 0.862). 

Figure 3. Means and 95-percent confidence intervals (CI) of the summated method performance scale (an eightitem Likert scale with Cronbach’s alpha = 0.86) for each of the dispute resolution method categories. Survey 

recipients were asked to rate the dispute resolution method used for eight different performance statements 

that evaluated whether the dispute resolution method improved different aspects of the dispute over science 

using a 5-point scale ranging from –2 (“strongly disagree”) to +2 (“strongly agree”). The highest and lowest 

possible summated scores for these eight Likert items were +16 and –16, respectively.

 The four-way ANOVA for the summated method performance scale across all methods 

revealed that the main effect of primary professional training was not significant (F(4, 268) = 0.165, p = 

0.956), the main effect of supervisory position was not significant (F(1, 268) = 2.83, p = 0.094), and the 

main effect of number of years worked at Reclamation was not significant (F(4, 268) = 2.25, p = 0.064). 

Dispute Resolution Professionals (DRPs)

Objectives and Hypotheses

The main objectives were to determine (1) how often DRPs were utilized in the disputes over 

science in which Reclamation employees have been involved, (2) the most common affiliations of these 
21

DRPs, (3) whether the DRPs were helpful in resolving disputes over science, (4) whether DRPs were 

utilized more or less frequently with some of the dispute resolution method categories than others, and 

(5) whether DRPs were rated as having a more positive or negative effect when they were utilized in 

conjunction with the different dispute resolution method categories.

We predicted that respondents that had experience with “public education, data sharing, and 

results dissemination,” “active collaboration,” and “adaptive management” would be more likely to also 

have experience with the use of a DRP. The implementation of either method type typically requires the 

involvement of multiple types of parties (Ruell and others, 2010); thus, we predicted that there would be 

a greater likelihood that a negotiator or mediator would be involved because there were more parties 

involved in these disputes that could make the request for one. 

Summary of Findings

• The use of DRPs was not common in the disputes over science in which Reclamation employees 

had been involved. Less than 17 percent of respondents described disputes that involved a DRP. 

• DRPs were most frequently affiliated with an independent consulting firm, followed by another 

Federal agency.

• In respondents’ experiences, DRPs mostly had a neutral or a somewhat positive effect on the 

resolution process for disputes over science.

• As we predicted, DRPs were more likely to be involved in disputes where either of the method 

categories “public education, data sharing, and results dissemination” and “adaptive management” 

were also used than in conjunction with the other method types. Contrary to our prediction, “active 

collaboration” was not more likely to involve a DRP than the other method categories. 

• Respondents’ ratings of the effect of the DRP on the dispute resolution process did not significantly 

differ across the different dispute resolution method categories. 

Detailed Results

There were 17 percent (n = 57) of respondents that described disputes over science for which a 

DRP was used, 75 percent (n = 259) that described disputes that did not include a DRP, and 8 percent (n

= 29) that did not know if a DRP had been used in the dispute. The number and percentage of 

respondents that indicated the affiliation type of the DRP involved in the dispute over science are 

presented in table 7. 

Table 7. Sample sizes and proportions of responses that had experience with dispute resolution professionals with 

each of the different affiliation types. 

Dispute resolution professional affiliation type

Number of 

responses

Percent1

An independent consulting firm 12 21

Another Federal agency 7 12

Bureau of Reclamation 4 7 

A stakeholder group 2 4 

Academia 2 4 

I don’t know: 2 4 

A State agency 1 2 

1

The percentage of respondents (n = 30) within each affiliation type.
22

The majority of respondents rated the effect of the DRP on the 5-point “negative effect” (–2) to 

“positive effect” (+2) scale as somewhat positive (fig. 4) with the median and mode values equal to 1. 

Only one respondent indicated that they did not know the effect of the DRP on the dispute over science. 

Figure 4. How respondents rated the perceived effects of dispute resolution professionals on dispute resolution 

processes on a 5-point “negative effect” to “positive effect” scale. The percentages within each category of the 

effect scale do not include the “I don’t know” response (n = 56). 

DRPs were utilized more frequently in conjunction with some dispute resolution method types 

than others (Pearson’s χ2

(6, n = 316) = 17.28, p = 0.008, excluding “I don’t know” responses). Post hoc 

z tests revealed that DRPs were more likely to be involved in disputes where either of the method 

categories “public education, data sharing, and results dissemination” and “adaptive management” were 

also used than in conjunction with the other method types. 

Respondents’ rating of the effect of the DRP on the resolution process did not differ across the 

different dispute resolution method types used (Kruskall-Wallis χ2

(6, n = 57) = 4.00, p = 0.677). 

Outcomes of Disputes over Science

Objectives and Hypotheses

The main objectives were to determine (1) the most common outcomes of disputes over science, 

(2) whether any of the dispute resolution method categories have different dispute outcomes than the 

others, (3) whether the use of any dispute resolution method result in different dispute outcomes from 

EXPLANATION
23

disputes for which no method was used, and (4) whether differences in respondents’ experiences were 

associated with their professional background and experience. 

For the first objective, we expected to find that most of the disputes over science would still be 

ongoing or unresolved because the literature suggests that competing and often incompatible interests 

frequently underlie disputes over science during decision-making processes for limited natural 

resources, and these kinds of disputes are very difficult to resolve (Ozawa and Susskind, 1985; Nie, 

2003; Sabatier and others, 2005; Scholz and Stiftel, 2005). In addition, we asked respondents to think 

only of the most recent dispute over science in which they had been involved. Thus, more recent 

disputes are more likely to be ongoing than older disputes. We did not have any a priori predictions for 

the last three objectives.

Summary of Findings

• The majority of respondents described a dispute over science that was still ongoing. Twenty-eight

percent selected that “the dispute was resolved,” and even a smaller percentage selected “the dispute 

was not resolved and is not expected to be resolved.” 

• There was no association between the dispute resolution method type used and the outcomes of the 

disputes over science. 

• The use of a dispute resolution method did not have different dispute outcomes from disputes where 

no method was used. 

• In general, respondents that had worked at Reclamation for less than 5 years were more likely to 

indicate that disputes were ongoing and less likely to indicate that disputes were resolved, and those 

that had worked at Reclamation for greater than 20 years were less likely to indicate that disputes 

were ongoing and more likely to indicate that disputes were resolved. 

Detailed Results

Responses on the status of the disputes over science are described in figure 5. 

EXPLANATION

Figure 5. Proportions and sample sizes of responses in each of the dispute outcome categories. 
24

Proportions of responses within each of the dispute outcome categories (excluding “I don’t 

know” responses) did not significantly differ across the dispute resolution method categories (Pearson’s 

χ

2

(12, n = 319) = 13.31, p = 0.347). Proportions of responses within each of the dispute outcome 

categories (excluding “I don’t know” responses) did not significantly differ between those that used a 

dispute resolution method versus the disputes where no method was used (Pearson’s χ

2

(2, n = 319) = 

0.474, p = 0.789). 

A GZLM for the dispute outcomes revealed that the main effect of primary professional training 

was not significant (Wald χ

2

(4, n = 283) = 7.35, p = 0.118), and the main effect of supervisory position 

was not significant (Wald χ

2

(1, n = 283) = 2.12, p = 0.145). The reduced GZLM revealed that the main 

effect of number of years worked at Reclamation was significant (Wald χ

2

(4, n = 286) = 17.17, p = 

0.002). 

Post hoc z tests of the contingency table of decision outcomes by the number of years worked at 

Reclamation indicated that respondents that had worked for Reclamation for less than 5 years were 

more likely than other respondents to indicate that the dispute was still ongoing and less likely to 

indicate that the dispute was resolved (Pearson’s χ2(8, n = 286) = 20.72, p = 0.008). Furthermore, 

respondents that had worked for Reclamation for more than 20 years were less likely than other 

respondents to indicate that the dispute was still ongoing and more likely than expected to indicate that 

the dispute was resolved. 

Resolved Disputes over Science

Objectives and Hypotheses

The main objective was to determine whether the disputes over science that were resolved were 

more frequently resolved within Reclamation or outside of Reclamation. If disputes were resolved 

within Reclamation, then we tried to determine at what level they were resolved, and if disputes were 

resolved outside of Reclamation, then we tried to determine which party was responsible for resolving 

the dispute. We did not have any a priori predictions regarding these objectives. 

Summary of Findings

• Disputes were most frequently resolved within Reclamation at the level at which the disputes 

originated. 

• When resolved outside of Reclamation, disputes over science were most frequently resolved by 

another Federal agency.

Detailed Results

If resolved, the majority of respondents described disputes that were resolved within 

Reclamation (57 percent (n = 48)), rather than outside of Reclamation (33 percent (n = 28). Only 10 

percent (n = 8) of respondents selected “I don’t know.” If disputes were resolved within Reclamation, 

the majority of respondents described disputes that were resolved at the level at which they originated 

(69 percent (n = 33)). In 29 percent (n = 14) of responses, higher level officials in Reclamation had 

resolved the disputes over science, and 10 percent (n = 8) of responses indicated that they did not know 

where the disputes were resolved. If the disputes were resolved outside of Reclamation, respondents 

most frequently indicated that another Federal agency resolved the dispute (43 percent (n = 12)), 

followed by the courts, academia, and State agencies (each with 11 percent (n = 2)). Either two or only 
25

one respondent indicated that disputes were resolved by elected officials, a Consultant Review Board, or 

an international entity.

Outcomes of Decisions Hindered by Disputes over Science

Objectives and Hypotheses

The main objectives were to determine (1) the most common outcomes of the decisions that 

were hindered by the disputes over science, (2) whether any of the dispute resolution method categories 

have different decision outcomes than the others, (3) whether the use of any dispute resolution method 

result in different decision outcomes from disputes for which no method was used, and (4) whether 

differences in respondents’ experiences were associated with their professional background and 

experience. 

We expected to find that most of the decision processes that had been hindered by these disputes 

over science would still be ongoing or would have been called to a halt because the disputes over 

science would be sufficiently problematic if not resolved (Ozawa and Susskind, 1985; Nie, 2003; 

Sabatier and others, 2005; Scholz and Stiftel, 2005) that they would continue to hinder the decisionmaking processes. We also expected this because we asked respondents to respond to survey questions 

thinking of the most recent dispute over science that they had experienced and more recent disputes may 

not have had sufficient time to reach resolution. 

The fifth and final objective was to learn whether the final outcomes of disputes over science 

potentially influenced the final outcomes of the decisions that were hindered by them. We predicted that 

disputes over science that were not resolved or ongoing would be associated with decision processes 

that were either not resolved or ongoing.

Summary of Findings

• As expected, the majority of respondents indicated that the decision process that had been hindered 

by the dispute over science was still ongoing. Roughly 1 out of every 3 respondents indicated that 

“the decision was made,” and roughly 1 out of every 10 respondents indicated that “the decision has 

not been made and it is uncertain as to whether it will be made.” 

• There was no association between dispute resolution method type and the outcomes of the decisions 

that were hindered by the dispute over science. 

• The use of a dispute resolution method also did not have different dispute outcomes from using no 

method. 

• The reported outcomes of decisions that were hindered by the disputes over science were 

significantly associated with the number of years that respondents had worked at Reclamation. 

Respondents that had worked for Reclamation for more than 10 years were more likely than other 

respondents to indicate that the decision was made and those that had worked for Reclamation for 

fewer than 5 years were less likely. In addition, respondents that had worked for Reclamation for 

more than 10 years were less likely than other respondents to indicate that “the decision-making 

process is ongoing” and those that had worked for Reclamation for fewer than 5 years were more 

likely.

• The outcomes of the decisions that were hindered by the disputes over science were significantly 

associated with the outcomes of the disputes over science. As predicted, the likelihood that decisions 

were made was higher when the disputes over science had been resolved. When dispute resolution 

processes were ongoing, the decision-making processes were more likely to be ongoing as well. 
26

Also, when respondents selected “the dispute was not resolved and is not expected to be resolved,” 

they were more likely to indicate that “the decision was not made and it is uncertain as to whether it 

will be made.” 

Detailed Results

Responses on the status of the decision that was hindered by the dispute over science are 

described in fig. 6. 

Figure 6. Proportions and sample sizes of responses in each of the decision outcome categories.

Proportions of responses within each of the decision outcome categories (excluding “I don’t 

know” responses) did not significantly differ across the dispute resolution method categories (Pearson’s 

χ

2

(12, n = 325) = 4.61, p = 0.970). Proportions of responses within each of the decision outcome 

categories (excluding “I don’t know” responses) did not significantly differ between those that used a 

dispute resolution method versus the disputes where no method was used (Pearson’s χ2

(2, n = 325) = 

1.00, p = 0.606). 

A GZLM for the decision outcomes revealed that the main effect of primary professional 

training was not significant (Wald χ

2

(4, n = 289) = 4.93, p = 0.295), and the main effect of supervisory 

position was not significant (Wald χ

2

(1, n = 289) = 0.028, p = 0.868). A reduced GZLM revealed that 

the main effect of the number of years worked at Reclamation was significant (Wald χ

2

(4, n = 292) = 

20.17, p < 0.001). 

Post hoc z tests of the contingency table of decision outcomes by the number of years worked at 

Reclamation revealed that respondents that had worked for Reclamation for more than 10 years were 

more likely than expected to indicate that the decision was made and those that had worked for 

Reclamation for fewer than five years were less likely than expected (Pearson’s χ2(8, n = 292) = 25.04, 

p = 0.002). Furthermore, respondents that had worked for Reclamation for more than 10 years were less 

EXPLANATION
27

Outcomes of Decisions Hindered by Disputes over Science

likely than expected to indicate that the decision-making process is ongoing and those that had worked 

for Reclamation for fewer than five years were more likely than expected.

Proportions of responses within each of the decision outcome categories were significantly 

associated with proportions of responses within each of the dispute outcome categories (Pearson’s χ2(4, 

n = 314) = 166.46, p <0.001; fig. 7). 

Figure 7. Proportions and sample sizes for each of the categories of dispute outcomes within each of the decision 

outcome categories (excluding “I don’t know” responses).

Experiences with Multiple Disputes over Science and (or) Dispute Resolution Method Types

Objectives and Hypotheses

The main objectives were to determine (1) how commonly Reclamation employees in technical, 

scientific, and managerial positions have experience with the use of multiple dispute resolution method 

types and (2) whether differences in respondents’ experiences were associated with their professional 

background and experience. We did not have any a priori predictions regarding these objectives. 

Summary of Findings

• Half of the respondents indicated that they had experience with zero or only one dispute resolution 

method type. Of the remaining respondents, 22 percent had experience with three or more dispute 

resolution methods categories. 

EXPLANATION
28

• Respondents’ primary professional training was significantly associated with their probability of 

having experience with multiple dispute resolution method types. Respondents trained in 

“engineering” were more likely than expected to indicate that they had experience with zero or only 

one dispute resolution method type, and respondents trained in “biological sciences” were more 

likely than expected to indicate that they had experience with at least three dispute resolution 

method categories.

Detailed Results

Out of the 347 respondents that completed loop 1 of the survey, 50 percent (n = 173) of 

respondents completed only loop 1, 28 percent (n = 97) completed two loops of the survey, and 22 

percent (n = 77) completed all three loops of the survey. 

A GZLM for the number of loops completed revealed that the main effect of supervisory 

position was not significant when taking into account primary professional training (Wald χ

2

(1, n = 305) 

= 0.223, p = 0.636), and the main effect of years worked at Reclamation was not significant when taking 

into account primary professional training and supervisory position (Wald χ

2

(4, n = 305) = 4.37, p = 

0.359). A reduced GZLM revealed that the main effect of primary professional training was significant 

(Wald χ

2

(4, n = 308) = 18.34, p = 0.001). 

Respondents trained in “engineering” were more likely to complete just one loop than expected 

in post hoc tests, and respondents trained in “biological sciences” were less likely to fill out just one 

loop than expected and were more likely to fill out all three loops than expected in post hoc tests of the 

contingency table of survey loops completed by primary professional training (Pearson’s χ

2

(16, n = 308) 

= 21.17, p = 0.007). 

Evaluations of Dispute Resolution Resources for Reclamation Employees

Objectives and Hypotheses

The primary objective was to determine which of the proposed resources would be the most 

useful for Reclamation employees that become involved in disputes over science. We did not have any a 

priori predictions for this objective. 

Summary of Findings

• Two of the potential dispute resolution resource options were considered to be more useful than the 

others. 

• Respondents rated “a training class” and then “a bibliography of related literature” as the most 

useful of the five potential resources. 

Detailed Results

A Friedman test revealed significant differences in how respondents rated the five potential 

resources for helping Reclamation employees resolve disputes over science (Friedman’s χ2

(4, n = 243) = 

45.73, p < 0.001 excluding responses that selected “I don’t know” or “not applicable” for any of the five 

resources). Approximately 1 out of every 5 respondents rated “a training class” and then “a bibliography 

of related literature” as “very useful” (a scale value of 5), and approximately 1 out of 10 respondents 

rated these resources as “not useful” (a scale value of 1; table 8). In contrast, the resource “an Indefinite 

Delivery/Indefinite Quantity (IDIQ) Contract for conflict services” was rated as the least useful of the 

five potential resources. Approximately 1 out of 10 respondents rated this resource as “very useful,” and 
29

1 out of 5 respondents rated this resource as “not useful.” A small percentage of the respondents gave 

“not applicable” (range 1–2 percent of respondents) or “I don’t know” answers (range 6–16 percent of 

respondents) for each of the factors. 

The distribution of respondents’ ratings did not significantly differ between the dispute 

resolution method categories for any of the potential dispute resolution resources. 

Table 8. Sample sizes, medians, modes, proportions of responses within each category of the 5-point “not useful” 

to “very useful” scale, and Friedman’s mean rank values for each of the different potential resources intended to 

help Reclamation employees resolve disputes over science. 

Potential resources to help 

Reclamation employees resolve 

disputes over science

Number of 

responses1 Median Mode

Scale

Not useful to very useful Freidman’s 

mean rank2

1 2 3 4 5

(1) Bibliography of related 

literature 290 4 4 10% 16% 23% 28% 22% 3.22

(2) Training class 289 4 4 9% 15% 25% 29% 22% 3.34

(3) Reclamation manual on 

managing disputes over science 284 3 3 15% 19% 29% 24% 13% 2.88

(4) Web site or SharePoint site 

devoted to managing disputes 

over science

278 3 4 14% 18% 27% 29% 13% 2.98

(5) Indefinite 

Delivery/Indefinite Quantity 

(IDIQ) Contract for conflict 

services

255 3 3 20% 21% 30% 17% 13% 2.58

1

The number of responses for each resource and the calculated medians, modes, and percentages within each category in the 

5-point not contested to highly contested scale do not include the "not applicable" and "I don't know" responses for that 

resource. 

2

Calculated from the responses (n = 243) that did not select "I don't know" or "not applicable" for any of the five resources. 

Acknowledgments 

We would like to thank Curtis A. Brown and Douglas R. Clark (U.S. Bureau of Reclamation) for 

their assistance with this project, Anthony Everette and Holly Miller (USGS Fort Collins Science 

Center) for their technological assistance with the Web survey, and Shannon Conk for testing the survey 

instrument. We also thank the individuals who completed the survey. This research was supported by 

funding from the Bureau of Reclamation, Agreement No. R11PG80819. 

References Cited 

AAPOR (American Association for Public Opinion Research), 2009, Standard definitions—Final 

dispositions of case codes and outcome rates for surveys (6th ed.): Ann Arbor, Mich., AAPOR, 

accessed December 27, 2010, at http://www.aapor.org/Standard_Definitions/2852.htm). 

AAPOR (American Association for Public Opinion Research), 2010, Response rate calculator, v. 3.1: 

Ann Arbor, Mich., AAPOR, accessed December 27, 2010, at 

http://www.aapor.org/Standard_Definitions/2852.htm). 
30

Abrams, N.E., and Berry, R.S., 1977, Mediation—A better alternative to science courts: Bulletin of the 

Atomic Scientists, April, p. 50–53. 

Adler, P.S., Barrett, R.C., Bean, M.C., Birkhoff, J.E., Ozawa, C.P., and Rudin, E.B., 2001, Managing 

scientific and technical information in environmental cases—Principles and practices for mediators 

and facilitators: Tucson, Ariz., U.S. Institute for Environmental Conflict Resolution, and Western 

Justice Center Foundation, 76 p., available at http://www.mediate.com/articles/envir_wjc11.pdf. 

Anderson, J.F., and Bingham, L., 1997, Upstream effects from mediation of workplace disputes: Some 

preliminary evidence from the USPS: Labor Law Journal, v. 48, p. 601–615. 

Baruch, Y., and Holtom, B.C., 2008, Survey response rate levels and trends in organizational research: 

Human Relations, v. 61, no. 8, p. 1139–1160. 

Becher, T., 1994, The significance of disciplinary differences: Studies in Higher Education, v. 19, no. 2, 

p. 151–161. 

Bowersox, J., 2000, From water development to water management—Federal agency opportunism in an 

era of policy devolution: American Behavioral Scientist, v. 44, no. 4, p. 599–613. 

Burkardt, N., Lamb, B.L., Taylor, J.G., and Waddle, T.J., 1995, Technical clarity in inter-agency 

negotiations—Lessons from four hydropower projects: Water Resources Bulletin, v. 31, no. 2, 

p. 187–198. 

Coglianese, C., 1999, The limits of consensus: Environment, v. 41, no. 3, p. 28–33. 

Dillman, D.A., Smyth, J.D., and Christian, L.M., 2009, Internet, mail, and mixed-mode surveys: The 

tailored design method: Hoboken, N.J., Wiley, 499 p. 

Downey, R.G., and King, C.V., 1998, Missing data in Likert ratings—A comparison of replacement 

methods: The Journal of General Psychology, v. 125, no. 2, p. 175–191. 

Ehrmann, J.R., and Stinson, B.L., 1999, Joint fact-finding and the use of technical experts, in Susskind, 

L., McKearnan, S., and Thomas-Larmer, J., eds., The consensus building handbook: A comprehensive 

guide to reaching agreement: Thousand Oaks, Calif., Sage Publications, p. 375–399. 

Endangered Species Act (ESA), 1973,16 U.S.C. § 1531 et seq. 

Hanemann, M., and Dyckman, C., 2009, The San Francisco Bay-Delta—A failure of decision-making 

capacity: Environmental Science & Policy, v. 12, p. 710–725. 

Henry, C.J., and Conrad, J.W., Jr., 2008, Scientific and legal perspectives on science generated for 

regulatory activities: Environmental Health Perspectives, v. 116, no. 1, p. 136–141. 

Ison, R., Röling, N., and Watson, D., 2007, Challenges to science and society in the sustainable 

management and use of water—Investigating the role of social learning: Environmental Science & 

Policy, v. 10, p. 499–511. 

Johnson, B.L., 1999, The role of adaptive management as an operational approach for resource 

management agencies: Conservation Ecology, v. 3, no. 2, p. 8., available at 

http://www.consecol.org/vol3/iss2/art8/. 

Koehler, B., and Koontz, T.M., 2008, Citizen participation in collaborative watershed partnerships: 

Environmental Management, v. 41, p. 143–154. 

Koontz, T.M., and Thomas, C.W., 2006, What do we know and need to know about the environmental 

outcomes of collaborative management: Public Administration Review, v. 66, p. 111–121. 

Koontz, T.M., Steelman, T.A., Carmin, J., Korfmacher, K.S., Moseley, C., and Thomas, C.W., 2004, 

Collaborative environmental management—What roles for government?: Washington, D.C.,

Resources for the Future Press, 200 p. 

Langfeldt, L., 2006, The policy challenges of peer review—Managing bias, conflict of interests and

interdisciplinary assessments: Research Evaluation, v. 15, no. 1, p. 31–41. 
31

Lorie, M.A., and Cardwell, H.E., 2006, Collaborative modeling for water management: Southwest 

Hydrology, v. 5, no. 4, p. 26–27. 

Mazur, A., 1973, Disputes between experts: Minerva, v. 11, p. 243–262. 

NRC (National Research Council), 2004, Confronting the nation’s water problems—The role of 

research: Washington, D.C., The National Academy Press, 310 p. 

National Environmental Policy Act (NEPA), 1969,42 U.S.C. § 4321 et seq. 

Nie, M., 2003, Drivers of natural resource-based political conflict: Policy Sciences, v. 36, p. 307–341. 

Ozawa, C.P., 1996, Science in environmental conflicts: Sociological Perspectives, v. 39, no. 2, 

p. 219–230. 

Ozawa, C.P., 2005, Putting science in its place, in Scholz, J.T., and Stiftel, B., eds., Adaptive 

governance and water conflict—New institutions for collaborative planning: Washington, D.C., 

Resources for the Future Press, p. 185–195. 

Ozawa, C.P., and Susskind, L., 1985, Mediating science-intensive policy disputes: Journal of Policy 

Analysis and Management, v. 5, no. 1, p. 23–39. 

Pisani, D.J., 2003, Federal reclamation and the American West in the Twentieth century: Agricultural 

History, v. 77, no. 3, p. 391–419. 

Quirk, P.J., 2005, Restructuring State institutions—The limits of adaptive leadership, in Scholz, J.T., 

and Stiftel, B., eds., Adaptive governance and water conflict—New institutions for collaborative 

planning: Washington, D.C., Resources for the Future Press, p. 204–212. 

Roberts, M.J., Thomas, S.R., and Dowling, M.J., 1984, Mapping scientific disputes that affect public 

policymaking: Science, Technology, and Human Values, v. 9, no. 1, p. 112–122. 

Roeder, E., 2005, Aquifer storage and recovery—Technology and public learning, in Scholz, J.T., and 

Stiftel, B., eds., Adaptive governance and water conflict—New institutions for collaborative planning: 

Washington, D.C., Resources for the Future Press, p. 106–118. 

Ruckelshaus, M.H., Levin, P., Johnson, J.B., and Kareiva, P.M., 2002, The Pacific salmon wars—What 

science brings to the challenge of recovering species: Annual Review of Ecological Systems, v. 33, 

p. 665–706. 

Ruell, E., Burkardt, N., and Clark, D.R., 2010, Resolving disputes over science in natural resource 

agency decision making: U.S. Bureau of Reclamation, Technical Services Center Technical 

Memorandum 86-68211-10-01, 58 p. 

Sabatier, P.A., Focht, W., Lubell, M., Trachtenberg, Z., Vedlitz, A., and Matlock, M., 2005, 

Collaborative approaches to watershed management, in Sabatier, P.A., Focht, W., Lubell, M., 

Trachtenberg, Z., Vedlitz, A., and Matlock, M., eds., Swimming upstream—Collaborative approaches 

to watershed management: Boston, Mass., MIT Press, p. 3–22. 

Schmidtz, D., 2000, Natural enemies—An anatomy of environmental conflict: Environmental Ethics, v. 

22, p. 397–408. 

Scholz, J.T., and Stiftel, B., 2005, The challenges of adaptive governance, in Scholz, J.T., and Stiftel, 

B., eds., Adaptive governance and water conflict—New institutions for collaborative planning: 

Washington, D.C., Resources for the Future Press, p. 1–11. 

Shih, T., and Fan, X., 2008, Comparing response rates from web and mail surveys—A meta-analysis: 

Field Methods, v. 20, no. 3, p. 249–271. 

Shinn, G., Baker, M., and Briers, G., 2007, Response patterns—Effect of day of receipt of an e-mailed 

survey instrument on response rate, response time, and response quality: Journal of Extension, v. 45, 

no. 2, article number 2RIB4. 
32

Stiftel, B., and Scholz, J.T., 2005, Conclusions—The future of adaptive governance, in Scholz, J.T., and 

Stiftel, B., eds., Adaptive governance and water conflict—New institutions for collaborative planning: 

Washington, D.C., Resources for the Future Press, p. 224–238. 

Ury, W.L., Brett, J.M., and Goldberg, S.B., 1988, Getting disputes resolved—Designing systems to cut 

the costs of conflict: San Francisco, Calif., Jossey-Bass Inc., Publishers, 201 p. 

van de Wetering, S.B., and McKinney, M., 2006, The role of mandatory dispute resolution in Federal 

environmental law—Lessons from the Clean Air Act: Journal of Environmental Law and Litigation, 

v. 21, p. 1–46. 

von Meier, A., 1999, Occupational cultures as a challenge to technological innovation: IEEE 

Transactions on Engineering Management, v. 46, no. 1, p. 101–114. 

Wagenet, L.P., and Pfeffer, M.J., 2007, Organizing citizen engagement for democratic environmental 

planning: Society and Natural Resources, v. 20, p. 801–813. 

Yun, G.W., and Trumbo, C.W., 2000, Comparative response to a survey executed by post, e-mail, and 

Web form: Journal of Computer-Mediated Communication, v. 6, no. 1, p. 0. doi: 10.1111/j.1083-

6101.2000.tb00112.x. 
33

Appendixes 
34

Appendix 1

Web Survey Questions

Unit 1

Question 1

Branch 1a

If respondents selected “no” in question 1, they were sent to the final unit 10 of the survey (professional 

background questions) before they submitted their surveys. 

Branch 1b

If respondents selected “yes” in question 1, they were sent to question 2. 

Question 2
35

Branch 2a

If respondents selected “no” or “I don’t know” in question 2, they were sent on to unit 3. 

Branch 2b

If they selected “yes” in question 2, they were sent on to unit 2. 

Unit 2 (Beginning of Loop 1)

Question 1

Unit 3 (Beginning of Loops 2 and 3)

Comment 1

Question 1
36

Comment 2 

Question 2

Branch 1a

Respondents who had answered “yes” to question 2 in unit 1 were now sent on to unit 4. 

Branch 1b

Respondents who answered “no” or “I don’t know” to question 2 in unit 1 were now sent on to unit 5. 
37

Unit 4

Question 1

Comment 1

Unit 5

Question 1

Branch 1a

Respondents that answered “yes” to question 1 were sent to question 2. 
38

Branch 1b

Respondents that selected “no” or “I don’t know” were sent on to unit 6. 

Question 2

Question 3

Unit 6

Question 1

Branch 1a

If respondents selected “the dispute was resolved” in question 1, they were sent to question 2. 

Branch 1b

If respondents selected any response option other than “the dispute was resolved” in question 1, they 

were sent on to unit 7. 
39

Question 2

Branch 2a

If respondents selected “the dispute was settled within Reclamation” in question 2, they were sent to 

question 3. 

Branch 2b

If respondents selected “parties outside of Reclamation decided the outcome” in question 2, then they 

were sent to question 4. 

Branch 2c

If respondents selected “I don’t know,” they were sent to unit 7. 

Question 3 

Question 4
40

Unit 7

Question 1

Comment 1

Branch 1a

If respondents had answered in unit 1 that “no, I have not been involved in the use of any of these 

methods,” they were now sent on to unit 9. 

Branch 1b

If respondents had previously selected and described one of the categories of methods in unit 1 and they 

had only completed one or two loops of the survey thus far, they were sent to unit 8. 

Branch 1c

If respondents had already completed three loops of the survey, they were sent on to unit 9. 

Unit 8

Question 1

Respondents were asked to select the dispute resolution method category with which they had the most 

experience from the list of remaining categories defined in unit 1 (this question was almost identical to 
41

question 1 in unit 2, except it excluded the method type(s) that were selected in previous loops and now 

included the response option “none of the above”). 

Branch 1a

If respondents to question 2 selected one of the remaining method categories, they were sent back to 

unit 3 (beginning of loop 2 and loop 3). 

Branch 1b

If respondents to question 2 answered “none of the above,” they were sent to unit 9. 

Unit 9

Comment 1

Question 1

Comment 2
42

Unit 10

Question 1

Question 2

Question 3

Respondents were then directed to submit their survey.
43

Appendix 2

Analyses of Survey Responses

The minimum response rate, which is labeled Response Rate 1 in Standard Definitions: Final 

Dispositions of Case Codes and Outcomes Rates for Surveys, 6th Edition (AAPOR, 2009), was 

calculated using the Response Rate Calculator V3.1 (AAPOR, 2010). The minimum response rate is a 

conservative estimate that is comparable across survey methods (Shih and Fan, 2008). 

All subsequent analyses were completed using IBM® SPSS® Statistics Desktop V20.0.0 (IBM 

Corporation, Armonk, New York, U.S.A.). 

Respondents’ Professional Background and Experience

We jointly coded all of the “other, please specify” responses to the question about respondents’ 

primary professional training and either put them into the categories that were provided or into one of 

the following additional categories: “technical trades and visitor services,” “architecture,” “computer 

science,” and “law enforcement.” We also combined the categories of “administration” and “business” 

and those that had indicated that their training was in realty or contracts into one category. We 

calculated frequencies of responses for each category of primary professional training, supervisory 

position, and the number of years employees had worked at Reclamation. 

Type I analyses were used for all GZLM analyses because the model predictors would most 

likely have sequential effects, and we could only really interpret their main effects because two of the 

variables had five or more categories. Primary professional training would almost always have come 

before supervisory position was determined, and both of these would have occurred (or have not 

occurred) before the survey when respondents’ tallied the total number of years they had worked at 

Reclamation thus far.

We used a GZLM with a multinomial distribution and a cumulative complementary log-log link 

function to evaluate the main effects of primary professional training (engineering, biological sciences, 

physical sciences, social sciences, administration/business/contracts/realty, law, technical trades/visitor 

services, architecture, computer science, law enforcement, other) and supervisory position (no, yes) on 

the number of years respondents’ had worked at Reclamation (0–5 years, 6–10 years, 11–15 years, 16–

20 years, >20 years). 

Involvement in Disputes over Science

We used a GZLM with a binomial distribution and logit link function to evaluate the main 

effects of primary professional training (engineering, biological sciences, physical sciences, social 

sciences, administration/business/contracts/realty, law, technical trades/visitor services, architecture, 

computer science, law enforcement, other), supervisory position (No, Yes), and the number of years 

worked at Reclamation (0–5 years, 6–10 years, 11–15 years, 16–20 years, >20 years) on respondents’ 

probability of having been involved in a dispute over science (No, Yes). 

We identified which primary professional training categories were more or less likely than the 

others to have been involved in a dispute over science using Pearson’s Chi-square test to compare the 

probability of being involved in a dispute versus the different primary professional training categories 

and using post hoc z tests (α = 0.05 using a Bonferroni correction for multiple tests). 
44

Dispute Characteristics

We only included survey responses to questions in loop 1 of the survey in subsequent analyses 

on question units 1–7 and excluded information obtained in loops 2 and 3 because it became apparent 

from the open-ended comments that neither the disputes over science nor the methods used in the 

dispute were independent across survey loops. Furthermore, because some of the respondents could 

have described the same dispute over science in loop 1 of their surveys (thus, disputes and dispute 

resolution attempts were not independent across respondents), all of the following results represent 

frequencies of respondents’ experiences rather than the actual frequencies of disputes and dispute 

resolution attempts. Also, we only included the incomplete surveys of respondents that had been 

involved in a dispute over science and had also answered the survey questions about the characteristics 

of the dispute and the performance questions (survey question units 1–7) in further analyses with the 

completed surveys. 

We jointly coded all of the open-ended comments that described the scientific issues behind the 

disputes over science into general categories of issues. More than one issue category per dispute was 

possible. We only coded a dispute as involving a Federally listed species if the respondent clearly 

identified it as such. Comments that mentioned that plant or animal species or species’ requirements 

were involved but did not give the species name or the Federal protection status of the species were 

coded simply as “native species” (unless they were identified as a non-native or invasive species), 

because we did not have enough information to identify whether or not a Federally listed species under 

the ESA (1973) was involved. We preferred to underestimate rather than overestimate the frequency of 

responses that involved Federally listed species.

We calculated the number of different issue types involved per response and calculated the range 

and proportion of responses that included one, two, three, or four issue types. We then calculated the 

proportions of responses for each of the underlying issue types (respondents could select all that apply). 

We calculated the number of different party types involved per response and calculated the 

mean, standard deviation, and range for the number of party types across respondents. We then 

calculated the proportions of responses for each of the outside party types that were involved 

(respondents could select all that apply). 

Dispute Resolution Methods

We calculated the proportions of responses that selected each of the dispute resolution method 

categories (respondents could only select one). 

Number of Issue and Party Types Involved in Dispute Versus Method Types Used

We compared the probability that one, two, three, or four different issue types were involved for 

each dispute resolution method category (including disputes for which no method was used) using 

Pearson’s Chi-square tests. 

We compared the mean number of outside party types involved per response between the dispute 

resolution method categories (including disputes for which no method was used) using a one-way 

ANOVA. We also compared the mean number of party types involved for responses where no dispute 

resolution process was used versus those where a dispute resolution process had been used using an 

independent-samples t test. 
45

Contested Factors in Disputes over Science

We calculated the median, mode, and proportions for each category in the ordered-category 

scale, which described how contested each factor was in the dispute. We assessed which of the seven 

potentially contested factors was rated as being the most or least contested using a Friedman test and by 

comparing the median and mode for each factor. We compared how respondents rated the degree to 

which each of the seven potentially disputed factors were contested between the dispute resolution 

method categories (including disputes for which no method was used) using Kruskal-Wallis one-way 

ANOVAs. These seven ordered-category items were not summed and analyzed using parametric 

statistics, because the items did not clearly form a meaningful construct when combined. 

Evaluations of Performance of Dispute Resolution Method Types

For each method type, we assessed whether respondents rated any of the eight performance

statements (unit 4, question 1) differently from the others and calculated the relative rank of statements 

using Friedman tests.

We combined the eight Likert items into a single summated method performance scale (that is, a 

Likert scale) in order to evaluate whether any of the method types were given better or worse overall 

agreement scores than the others. This eight-item Likert scale demonstrated sufficient internal 

consistency (Cronbach’s alpha = 0.86). We excluded the 15 responses that provided answers for fewer 

than four of the Likert items. After excluding these responses, 72 (24 percent) responses and 124 (5 

percent) items contained missing values, which included the “I don’t know” and “not applicable” 

responses as well as the refusals. We replaced these missing values with the mean value for the item 

across all respondents because using the individual respondent mean value could have artificially 

inflated inter-item correlations (Downey and King 1998). The estimated Cronbach’s alpha for these 

eight items remained the same after these replacements. We calculated means and 95-percent

confidence intervals for the summated method performance scale and compared respondents’ evaluation 

of the different dispute resolution method categories using a one-way ANOVA. 

We used a four-way ANOVA to evaluate the main effects of primary professional training 

(engineering, biological sciences, physical sciences, social sciences, 

administration/business/contracts/realty), supervisory position (no, yes), and years worked at 

Reclamation (0–5 years, 6–10 years, 11–15 years, 16–20 years, >20 years) on respondents’ score on the 

summated method performance scale. We excluded the other four primary professional training 

categories in this analysis because they had two or fewer responses.

Dispute Resolution Professionals (DRPs)

We calculated the median, mode, and the proportions of responses within each of the DRP 

affiliation types, and proportions within each category in the ordered-category scale that described the 

effect of the DRP on the resolution process. We compared the likelihood that respondents were more or 

less likely to report that DRPs had been used in conjunction with any of the dispute resolution method 

categories using a Pearson’s Chi-square test and post hoc z tests (α = 0.05 using a Bonferroni correction 

for multiple tests). We compared whether respondents rated the effect of the DRP differently depending 

on which dispute resolution method types were also used for the dispute using a Pearson’s Chi-square 

test because some of the method categories had fewer than five responses.
46

Outcomes of Disputes over Science

We calculated the proportions of responses within each of the dispute outcome categories (“the 

dispute was resolved,” “the dispute resolution process is ongoing,” “the dispute was not resolved and is 

not expected to be resolved,” “I don’t know”). 

The proportions of responses within each of the dispute outcome (excluding “I don’t know” 

responses) were compared across the dispute resolution method categories using a Pearson’s Chi-square 

test. The proportions of responses within each of the dispute outcome categories (excluding “I don’t 

know” responses) were compared between disputes where a dispute resolution method had been used on 

disputes where no method was used using a Pearson’s Chi-square test.

We used a GZLM with a multinomial distribution and logit link function to evaluate the main 

effects of primary professional training (engineering, biological sciences, physical sciences, social 

sciences, administration/business/contracts/realty), supervisory position (no, yes), and years worked at 

Reclamation (0–5 years, 6–10 years, 11–15 years, 16–20 years, >20 years) on dispute outcome (the 

dispute was resolved, the dispute is ongoing, the dispute was not resolved and is unlikely to be resolved 

in the future). We excluded the other four primary professional training categories in this analysis 

because they had two or fewer responses. Because the main effect of primary professional training was 

nonsignificant, and the main effect of supervisory role was nonsignificant when taking into account 

primary professional training, we re-ran a reduced GZLM without these variables. 

We identified which categories of the number of years worked at Reclamation were more or less 

likely than expected to select the different decision outcomes using a Pearson’s Chi-square test and post 

hoc z tests (α = 0.05 using a Bonferroni correction for multiple tests).

Resolved Disputes over Science

We calculated the proportion of responses within each of the dispute resolution location 

categories (within Reclamation, outside of Reclamation; I don’t know). 

Outcomes of Decisions Hindered by Disputes over Science

We calculated the proportion of responses within each of the decision outcome categories (the 

decision was made, the decision-making process is ongoing, the decision was not made and it is 

uncertain as to whether it will be made, I don’t know). 

We used a GZLM with a multinomial distribution and logit link function to evaluate the main 

effects of primary professional training (engineering, biological sciences, physical sciences, social 

sciences, administration/business/contracts/realty), supervisory position (no, yes), and the number of 

years worked at Reclamation (0–5 years, 6–10 years, 11–15 years, 16–20 years, >20 years) on the 

decision outcome (the decision was made, the decision-making process is ongoing, the decision was not 

made and is unlikely to be made in the future). We excluded the other four primary professional training 

categories in this analysis because they had two or fewer responses. Because the main effect of primary 

professional training was nonsignificant, and the main effect of supervisory role was nonsignificant 

when taking into account primary professional training, we re-ran a reduced GZLM without these 

variables.

We identified which categories of the number of years worked at Reclamation were more or less 

likely than expected to select the different decision outcomes using a Pearson’s Chi-square test and post 

hoc z tests (α = 0.05 using a Bonferroni correction for multiple tests).

The proportions of responses within each of the decision outcome categories (excluding “I don’t 

know” responses) were compared across the dispute resolution method categories using a Pearson’s 
47

Chi-square test. The proportions of responses within each of the decision outcome categories (excluding 

“I don’t know” responses) were compared between disputes where a dispute resolution method had been 

used versus disputes where no method was used using a Pearson’s Chi-square test.

Experiences with Multiple Disputes over Science and (or) Dispute Resolution Methods

We calculated the proportion of respondents that completed just one loop, two loops, or all three 

loops of the survey. 

We used a GZLM with a multinomial distribution and logit link function to evaluate the main 

effects of primary professional training (engineering, biological sciences, physical sciences, social 

sciences, administration/business/contracts/realty), supervisory position (no, yes), and the number of 

years worked at Reclamation (0–5 years, 6–10 years, 11–15 years, 16–20 years, >20 years) on the 

number of survey loops completed (one, two, three). We excluded the other four primary professional 

training categories in this analysis because they had two or fewer responses. Because the main effect of 

supervisory role was nonsignificant when taking into account primary professional training, and the 

main effect of number of years worked at Reclamation was nonsignificant when taking into account 

primary professional training and supervisory role, we re-ran a reduced GZLM without these variables.

We identified which primary professional training categories were more or less likely than 

expected to complete one, two, or three loops of the survey using a Pearson’s Chi-square test and post 

hoc z tests (α = 0.05 using a Bonferroni correction for multiple tests).

Evaluations of Dispute Resolution Resources for Reclamation Employees

We calculated the median, mode, and the proportions of responses within each category in the 

ordered-category scale, which described the usefulness of each potential resource. We assessed which of 

the five potential resources respondents rated as being the most or least useful using a Friedman test and 

by comparing the median and mode for each factor. 
48

Appendix 3

Glossary of Analyses

Analysis of variance (ANOVA). A parametric statistical test of whether or not the means of different 

groups are equal. An ANOVA is a general linear model that assumes that cases are independent, the 

distribution of the residuals is normal, and that the groups’ variances are equal. Non-normal, continuous 

data can often be transformed (that is, square-root transformation) so that it has a distribution that more 

closely approximates normal.

Bonferroni correction. A correction of the alpha level used for multiple significance tests in order to 

reduce the probability of false positives (that is, to reduce type I error).

Cronbach’s alpha. An index of reliability or internal consistency across items (that is, questions, 

rankings, ratings, and so forth) for a sample. A higher value indicates that the test results have higher 

reliability.

Friedman test. A nonparametric statistical test that compares the distributions of ranks of repeated 

measures on a single sample.

Generalized linear model (GZLM). A generalization of ordinary least squares regression for non-normally 

distributed data that uses a link function to relate the linear model to the response variable. These tests 

are commonly used for binomial, ordinal, and count data, which do not follow the assumptions of 

general linear models. It is common practice that when the variation explained by one or more variables 

in the original model are not statistically significant, reduced models can be subsequently run that 

exclude these variables.

Independent-samples t test. A statistical test that tests whether or not the means of two normally 

distributed independent samples are equal. Non-normal, continuous data can often be transformed (that 

is, square-root transformation) so that it has a distribution that more closely approximates normal.

Kruskall-Wallis one-way analysis of variance. A nonparametric statistical test that compares the 

distribution of ranks across independent samples. A Kruskall-Wallis one-way ANOVA does not assume 

that the population approximates normal but does assume an identical scale and shape to the groups’ 

distributions. 

Likert item. A statement or question for which the respondent is asked to rank along a symmetric scale 

that depicts a range of intensity, such as agree to disagree. Likert items are often used with 5-point, 7-

point, or 9-point ordered-category scales with a neutral middle value.

Likert scale. A sum of responses to multiple Likert items or questions (that is, a summated scale). Likert 

scales are commonly used in survey research.

Pearson’s chi-square test. A statistical test of independence that tests whether the frequency of paired 

observations of two categorical variables differs from what would be expected if the variables were 

independent. Post hoc tests (that is, Tukey or z tests) can be used when one or both of the categorical 

variables have more than two categories to determine which pairs of cells in the contingency table are 

significantly different. Generally a Bonferroni correction is used with post hoc tests in order to reduce 

the probability of false positives. 
49

Publishing support provided by: 

Denver Publishing Service Center

For more information concerning this publication, contact:

Center Director, USGS Fort Collins Science Center

2150 Centre Ave., Bldg. C

Fort Collins, CO 80526-8118

(970) 226-9398

Or visit the Fort Collins Science Center Web site at:

http://www.fort.usgs.gov/