Document ID: EPA-HQ-OGC-2007-0982-0010
Agency: epa
Document Type: Supporting & Related Material
Title: 
Posted Date: 2009-04-27T04:00Z

Part B of the Supporting Statement

1.	Survey Objectives, Key Variables, And Other Preliminaries

	(a)	Survey Objectives

	The objectives of the survey are (1) to identify the environmental and
economic results, and decision-making process efficiency of
environmental conflict resolution (ECR) decision-making processes; and
(2) to compare the results and decision-making process efficiency of ECR
and non-ECR decision-making processes.

	(b)	Key Variables

	The primary focus of the survey is on the environmental and economic
effects of the decisions resulting from ECR decision-making processes. 
The survey will also be applied to a small number of non-ECR
decision-making process cases for purposes of comparison.  For both ECR
and non-ECR cases, the key variables include:  1) environmental effects
for the actual case and the likely alternative decision-making process;
and 2) economic effects, including the time and direct costs for the
actual case and the likely alternative decision-making process, and
changes in social capital among parties to the decision.  This
information enables evaluation of the efficiency and the costs and
benefits of ECR decision-making processes.

	(c)	Statistical Approach

	The survey will be administered as a one-time, voluntary data
collection using a combination of census and purposive sampling
approaches.  A census approach will be used to survey participants in
all ECR cases for identified EPA program areas (e.g., the Superfund
program) meeting the selection criteria (e.g., agreement-reached,
agreement being implemented, took place within a specified time frame). 
A census approach to ECR cases in EPA program areas is useful and
necessary because the Agency’s ECR program has operated without such a
survey (and the concomitant ability to conduct this evaluation research)
for many years and urgently needs program-by-program ECR results and
decision efficiency information to make recommendations on selecting
among ECR and other decision-making process options in different program
areas.  Purposive sampling will also be used for ECR cases on an as
needed basis.

	We will use a purposive sampling approach for non-ECR decision-making
processes, selecting non-ECR cases for comparison to ECR cases based on
characteristics they have in common, such as the number and types of
participants involved in the case (e.g., levels of government, types of
private sector entities) and type of environmental issues at stake.  The
purposive sampling approach to non-ECR cases is appropriate for this
evaluation research because the ECR cases in any EPA program area are
relatively small in number and are not likely to be as diverse as the
entire population of decision-making process cases in that program area.
 Thus, we only need to match and collect data on a limited set of
non-ECR cases to make the necessary comparison.

	EPA has engaged SRA International, Inc. (3434 Washington Blvd.,
Arlington, VA 22201) to help design, administer, and provide technical
assistance and support for the survey. Furthermore, the contractor will
assist in analyzing the data and preparing the draft report.

	(d)	Feasibility

	Based on previous pilot testing of the survey methodology in Oregon and
on a set of EPA cases, we anticipate little difficulty in achieving good
response rates from respondents.  In the pilot tests, response rates
were excellent – 75% and 88% in the Oregon and EPA cases,
respectively.  Respondents have been highly cooperative and have
encountered no difficulty providing the information requested.  The use
of web-based surveys and tailored questions (e.g., the name of the case,
specific environmental effects) has greatly enhanced the respondents’
ability to provide the data requested.

	EPA has sufficient funding to complete the survey and we do not
anticipate any future funding challenges.  Assuming timely approval by
OMB, we anticipate the results of the survey will be available for
timely use in administering the Agency’s ECR program.

2.	Survey Design

	(a)	Target Population and Coverage

	The survey will target ECR and non-ECR decision-making processes, which
will vary in number depending on the EPA program (e.g., Superfund) in
which they occurred.  The survey will be administered to all
participants who represented parties to ECR and non-ECR cases.

	(b)	Sample Design

	All participants representing parties to ECR and non-ECR cases
(selected through the statistical approaches described above) will be
invited to participate in the survey, thus no sampling is required.  We
anticipate an average of six respondents per case.

	(c)	Precision Requirements

	Based on previous pilot testing, we anticipate no concerns related to
precision.  In two pilot tests, we achieved high response rates of 75%
and 88%.  Analysis of respondent data showed no response bias. 
Responses for the key environmental measures were statistically reliable
and valid.

	(d)	Questionnaire Design

	The questionnaire design for this survey has benefited significantly
from earlier pre-testing and pilot tests, including direct feedback from
respondents.  We have been able to demonstrate robustness in a field
setting and establish question types, wording, and order that produce
the results necessary for this survey.  We have also reduced the burden
on respondents by eliminating extraneous questions from previous
versions.  Our experience is that one of the most critical design
elements for producing reliable results is the ability to tailor certain
questions (e.g., those relating to specific environmental effects) on a
case-by-case basis within an overall fixed questionnaire structure.

	The resulting questionnaires include three main types of questions that
will provide data to fulfill the survey objectives.  The first set of
questions (questions 1-18 and 31) concerns the decision-making process,
including the respondents’ views on the extent to which their
priorities were achieved, the quality of the process, the availability
of information in the process, and the likely alternative
decision-making process.  Results from these questions assist in
interpreting findings related to economic and environmental effects and
in making appropriate case comparisons.  A second set of questions
(questions 19-30 and 32-44) addresses the economic effects of the
decision-making process and the likely alternative decision-making
process, including time, direct costs, and generation of social capital.
 The final set of questions (questions 45-53) concerns environmental
effects (tailored to the specific case) for the decision-making process
and the likely alternative decision-making process.

3.	Pretests And Pilot Tests

	Prior applications of the survey methodology have provided ample
pretesting of the questionnaire and piloting the survey administration
procedures.  The pretesting and piloting efforts resulted in revisions
to address respondents’ suggestions for improvement and enhance the
quality of the survey and administration procedures.

4.	Collection Methods And Follow-up

	(a)	Collection Methods

	The survey will be administered electronically through a web-based
application.  This collection method was selected because of its
reliability, usability, precision, and cost effectiveness.  An advance
email from EPA will be sent to all parties, followed by another email
with the link to the survey.  Two to three additional email reminders
are sent, and the survey administrator telephones any remaining
non-respondents.

	(b)	Survey Response and Follow-up

	The target response rate is 75% overall and for each type of interest
represented in a case (e.g., tribal, industry, environmental group,
state agency).  Item non-response has been assessed and has not been a
problem in pilot testing; we expect questionnaires will be fully
completed.  If requested, a summary of the results will be provided to
respondents.

5.	Analyzing And Reporting Survey Results

	(a)	Data Preparation

	Web administration of the survey will provide electronic data free of
data entry errors.  The data will be carefully reviewed for any missing
responses; full responses are retained in the dataset.  Based on
previous pilot testing, missing data is not expected to be a significant
issue.  SRA International, Inc. will process the survey data under EPA
supervision.

	(b)	Analysis

	Analysis will be conducted on data from individual cases and across
multiple cases using standard statistical methods.  Categorical
variables will be cross-tabulated and analyzed using non-parametric
statistical tests of independence such as the chi square test.  We will
calculate measures of central tendency on scalar variables and compare
means where feasible.  Cost data for individual cases and their likely
alternative decision-making processes will be computed by adding
monetized time data (using standard hourly rates) and specified monetary
values.  Magnitude and probability data for environmental effects will
be computed into an index of environmental effects (with values from -1
to 1) so that environmental effects can be aggregated and compared
across cases.

	We will display survey results in a variety of ways depending on the
audience, its interests, and its level of methodological sophistication.
 Typical results presentations are likely to include charts for
individual and multiple cases displaying comparisons of environmental
effects indices, cost-effectiveness results in time and dollars, and
means for social capital scalar variables.

	(c)	Reporting Results

	This survey will enable EPA to evaluate the environmental and economic
effects of agreements reached through ECR processes compared to
decisions that might have been achieved through other decision-making
processes (e.g., litigation).  Survey results will be used to analyze
and assess the Agency’s ECR procedures, to ensure program activities
are executed and managed in a cost-effective manner, and provide
information for agency managers and staff to use when deciding whether
to use ECR, consistent with the principles of the Government Performance
Results Act and the President’s reform agenda.  For these reasons, the
primary audiences for the survey results are within EPA.  We will use
summary reports and briefings to provide information about the survey
results in this context.

	This survey will also help satisfy requirements of the November 2005
memorandum from OMB and CEQ that directed agencies, among other things,
to focus on accountable performance and achievement in ECR.  EPA will
provide summary results from this survey to OMB and CEQ as part of its
ECR annual report, which is required under the ECR policy memorandum. 
We will also provide a briefing for the quarterly ECR policy forum, a
group of senior agency officials convened by OMB and CEQ.

	Finally, we are aware that ECR consumers, practitioners, and
researchers are eagerly awaiting the results of this evaluation research
and have responded favorably to our survey approach.  We will
communicate survey results to these audiences through summary reports
and conference presentations.

 PAGE   

 PAGE   1