Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP

Quality Assurance Project Plan

Work Assignment No. 4-05, EPA Contract No. EP-C-07-023

Assessment of the Economic Costs and Impacts and Benefits of the Final 

316(b) Existing Facilities Rulemaking

Submitted by:

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138

(617) 492-7100

Submitted to:

Wendy Hoffman, Work Assignment Manager

and 

Damon Highsmith, Contract Project Officer

U.S. Environmental Protection Agency

Office of Science and Technology

Engineering and Analysis Division

1200 Pennsylvania Ave., NW

Washington, D.C. 20460

November 3, 2011

Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP

QUALITY ASSURANCE PROJECT PLAN (QAPP) FOR:

ASSESSMENT OF AND PLANNING FOR THE NECESSARY ECONOMICS AND 

BENEFITS WORK TO COMPLETE THE 316(B) RULEMAKING

Table of Contents

1 Project Objectives, Organization, and Responsibilities.............................................................1-1

1.1 Project Objective/Goal ...........................................................................................................1-1

1.2 Project requirements related to primary and secondary data..................................................1-2

1.3 Data analysis...........................................................................................................................1-3

1.4 Training and competency requirements necessary for key project personnel........................1-3

1.5 Role and responsibilities of project personnel........................................................................1-3

1.5.1 Project Management and Performance.......................................................................1-3

1.5.2 Summary of Quality Assurance Project Plan Responsibilities and 

Implementation...........................................................................................................1-4

2 Acquisition and Use of Primary Data..........................................................................................2-1

2.1 Clean Water Act Section 308 Survey of Electric Power Generating Facilities and 

Manufacturing Facilities.......................................................................................................2-1

2.2 316(b) Stated Preference Survey............................................................................................2-1

2.3 Selecting Survey Participants.................................................................................................2-2

2.4 Survey Printing and Mailing ..................................................................................................2-2

2.5 Data Quality and Data Entry ..................................................................................................2-2

2.6 Data Analysis .........................................................................................................................2-3

2.7 Conducting Non-Response Study...........................................................................................2-3

3 Acquisition and Use of Secondary Data ......................................................................................3-1

3.1 Sources used to identify and rationale for selection of secondary data..................................3-1

3.2 Quality requirements of and acceptance criteria for secondary data......................................3-2

3.3 Data Reduction, Validation, and Verification Procedures and Documentation .....................3-3

3.3.1 Data reduction procedures and documentation...........................................................3-3

3.3.2 Data validation and verification procedures and documentation................................3-3

3.3.3 Data storage ................................................................................................................3-5

4 Expected Work Products..............................................................................................................4-1

5 Acceptance by the EPA WAM .....................................................................................................5-1
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP i

Changes to QAPP for Support of Work Assignment 4-05 under Contract EP-C-07-023

Revision 0 1

Date February 22, 

2008November 3, 2011 September 16, 2011

1: Project Objectives, Organization, and Responsibilities

1.1: Project 

Objective/Goal NA Updated to reflect project objectives/goals as outlined in the work assignment 4-05, including a 316(b) stated preference survey. 

1.2: Project requirements 

related to primary and 

secondary data

NA Updated to reflect project objectives/goals as outlined in the work assignment 4-05, 

including 316(b) stated preference survey. 

1.3: Data analysis NA Non-substantive changes to the wording. Described suppressed/withheld data estimation in 

more detail; provided an example.

1.4: Training and 

competency requirements 

necessary for key project 

personnel

NA Section added in its entirety.

1.5: Role and 

responsibilities of project 

personnel

NA

Changes to reflect current project staffing/needs. Added more information on PQA’s role 

and QA process. Added information on interaction between Abt Associates and Abt SRBI 

during survey implementation.

2: Error! Not a valid result for table.

2.3: Selecting Survey 

Participants NA Section added in its entirety.

2.4: Survey Printing and 

Mailing NA Section added in its entirety.

2.5: Data Quality and Data 

Entry NA Section added in its entirety. 

2.6: Data Analysis NA Section added in its entirety.

2.7: Conducting NonResponse Study NA Section added in its entirety.

3: Acquisition and Use of Secondary Data

3.1: Sources used to 

identify and rationale for 

selection of secondary data

NA

Consolidated contents of Sections 2.1 and 2.2 from the current QAPP (Revision 

0).Updated to reflect current project needs. Added QA guidelines for accepting 

publications, working papers, etc. 

3.2: Quality requirements 

of and acceptance criteria 

for secondary data

NA Consolidated contents of Sections 2.3 and 2.4 from the current QAPP (Revision 0).

3.3: Data Reduction,

Validation, and 

Verification Procedures

and Documentation

NA

Added more detailed information on internal review, data storage, and documentation 

processes. Added a list of QA measures for written work product. Made non-substantive 

changes to the wording. Added Table 1.

4: Expected Work Products

4: Expected Work 

Products NA Updated to reflect current project needs.

5: Acceptance by the EPA WAM

5: Acceptance by the EPA 

WAM NA Non-substantive changes to the wording.

Appendix 1: Abt SRBI - Data Collection and Data Processing Quality Assurance Plan; 316(b) Survey Environmental 

Protection Agency

Appendix 1: Abt SRBI -

Survey NA Appendix added in its entirety. 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 1-1

1 Project Objectives, Organization, and Responsibilities

1.1 Project Objective/Goal

Under Contract EP-C-07-023, work assignment 4-05, Abt Associates supports the Office of Science and 

Technology/Engineering and Analysis Division in analyzing the cost and economic impacts, 

environmental effects, and economic benefits for the Revised Existing Facilities regulation under Section 

316(b) of the Clean Water Act (referred to in this document as the Revised 316(b) Existing Facilities Rule

or the Revised 316(b) Rule). In doing so, Abt Associates will support the Notice of Data Availability 

(NODA) and the final stages of rule development. Ongoing and future work in support of the Revised 

316(b) Existing Facilities Rule under this work assignment includes: 

 Review and assessment of analyses undertaken and data developed and presented in support of 

the Revised 316(b) Existing Facilities Proposed Rule

 Potentially revising the existing methodology and/or databases, and developing new analyses

 Implementing updated/revised analyses and data sources to support the Revised 316(b) Existing 

Facilities Rule

 Preparing summaries of new cost and economic impact analyses as well as new and innovative 

economic benefits and aquatic health analyses, conducting the stated preference survey, analyzing 

and incorporating survey results in the benefits analysis, and providing draft sections for the cost 

and economic impact, and environmental and economic benefits analyses parts of the NODA, 

respectively, eventually leading to the final draft version of the NODA report

 Supporting the development and assessment of regulatory options for the Revised 316(b) Existing 

Facilities Rule to determine the three to five primary regulatory options to be presented in the 

final Federal Register Preamble (Preamble) and Technical Support Document (TSD) report

 Conducting cost and economic impact as well as environmental and economic benefits analyses

for development of Preamble, Cost and Economic Impact Analysis TSD report, and 

Environmental and Economic Benefits TSD report

 Supporting preparation of the Preamble, Final Cost and Economic Impact Analysis TSD, and 

Environmental and Economic Benefits TSD materials required for publication of the Revised 

316(b) Existing Facilities Final Rule

As specified in work assignment 4-05, Abt Associates prepared and submits herewith a Quality 

Assurance Project Plan (QAPP) for the data acquisition and validation, analyses, and documentation

required to support the Revised 316(b) Existing Facilities Rule. This QAPP contains procedures that are 

consistent with EPA’s quality assurance guidance specified in EPA Requirements for Quality Assurance 

Project Plans (EPA, 2001). The QAPP was developed to meet the standards of accuracy and transparency 

for EPA regulatory analyses and will provide quality control for all primary and secondary data used 

under this work assignment. Abt Associates’ quality assurance coordinators and project management 

team plan, implement, and assess project quality procedures throughout all project levels. These 

procedures are tailored to the needs of each individual task and will be used on this work assignment. A 

copy of this QAPP will be provided to all Abt Associates staff involved in this project, including those 

who join the project after preparation of the QAPP.

All technical work pertinent to implementation of the 316(b) stated preference survey is performed by 

Abt Associates’ survey implementation unit (Abt-SRBI). Accordingly, Abt-SRBI prepared and submits

QAPP for data collection and entry performed under Task 5.1. The plan contains procedures that are 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 1-2

consistent with EPA’s quality assurance guidance and provides quality control for all primary data 

collection. The QAPP for the survey implementation and data entry is provided in Attachment 1.

1.2 Project requirements related to primary and secondary data

For analyzing the cost and economic impact of the Revised 316(b) Rule, Abt Associates will continue to 

rely on certain primary source data that were collected as part of the CWA § 308 survey of electric power 

generating facilities and manufacturing facilities for the original 316(b) Phase II and Phase III regulatory 

analyses. These data contain information on the design and operation of cooling water intake structures 

present at facilities expected to be subject to the Revised 316(b) Existing Facilities Rule. The 

questionnaire responses received in the earlier survey effort underwent substantial cleaning and quality 

validation at that time and we do not anticipate performing additional data cleaning and quality validation 

on these data for the current effort. Abt Associates will continue to use these data to support important 

elements of the analysis, particularly for the analysis of manufacturing facilities. 

In addition to using this primary source data from the 316(b) facility questionnaire, Abt Associates will 

also rely substantially on secondary source data for this analysis. Secondary source data for this analysis 

include, but are not limited to: 

 Data descriptive of current operating and business conditions at the electric power generating and 

manufacturing facilities expected to be subject to the Revised 316(b) Existing Facilities Rule, and 

the business conditions of the firms and other entities that own these facilities. These data will 

support the assessment of baseline operations and economic/financial performance for facilities 

subject to regulation, and thus provide the basis for understanding the impact of the regulation on 

these facilities and the firms that own them. 

 Data generally describing economic/financial conditions in, and the regulatory environment of, 

the electric power industry and manufacturing industries subject to the Revised 316(b) Existing 

Facilities Rule.

 Data on electricity prices, which may be affected by the regulation and thus in turn cause an 

economic impact on electricity consumers. 

 Cost and/or price index data that are used to adjust dollar values from one time period to another 

time period. 

 Information on facilities’ NPDES permitting schedules, which is in turn used to estimate 

facilities’ expected compliance schedules.

Abt Associates also expects to rely on analysis and outputs from the Integrated Planning Model (IPM) to 

assess the potential impact of regulatory requirements on electric power generating facilities and general 

market conditions within the context of regional and national electricity markets. 

For analyzing the benefits expected to be achieved by the Revised 316(b) Existing Facilities Rule, Abt 

Associates will rely on primary and secondary source data to characterize benefits of reducing 

impingement and entrainment. Under the previous work assignment (WA3-05), Abt Associates 

developed a stated preference survey to elicit total willingness-to-pay values for ecological improvements 

associated with reducing impingement and entrainment of aquatic species at CWIS. Section 2: Acquisition 

and Use of Primary Data provides details of data collection and analysis.

Secondary source data for the environmental and economic benefits analysis include but are not limited 

to:
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 1-3

 Data describing commercial landings and the value of commercial catch for each of the seven 

study regions. 

 Data describing recreational fishing in each of the seven study regions, including recreational 

catch by species, the value of recreational catch, and the number of participants.

 Data on the federally listed threatened and endangered species present in each analysis region that 

are affected by impingement and entrainment. 

 Data on facilities’ NPDES permitting schedules, which is used to estimate facilities’ expected

compliance schedule. 

Section 3: Acquisition and Use of Secondary Data provides details of the secondary data sources, 

including criteria for selection and use. Section 3 also discussed the selection of IPM to support cost and 

economic impact analysis.

1.3 Data analysis

Abt Associates will compile, analyze, and maintain data extracted for this work in Access databases,

Excel spreadsheets, or SAS datasets and related analytic systems. Abt Associates will conduct analyses 

using database queries, and will summarize data and analysis findings in databases, spreadsheets, and/or 

documents. All databases, spreadsheets, and documents will cite the origin of the data and the database 

version (if any). We will clearly document any modifications to the data, and will appropriately flag these 

modifications in the final data set to differentiate between values unchanged from the original sources and 

values that were estimated or modified for use in the analysis of the Revised 316(b) Existing Facilities 

Rule. In particular, where data have been suppressed/withheld in the original source database due to 

confidentiality concerns, we may need to estimate these suppressed/withheld data to ensure consistency 

and comprehensiveness of the modeling framework. For instance, in certain instances Economic Census 

data are withheld from publication because of a small number of observations in a particular economic 

sector and revenue range; in these instances, it may be necessary to (1) use data from adjacent revenue 

ranges or the total of revenue ranges to describe characteristics of the missing sector/revenue range 

combination, or (2) use data from a higher level of sector definition (e.g., NAICS 5-digit sector instead of 

6-digit sector) to describe characteristics of the missing data item.

1.4 Training and competency requirements necessary for key project personnel

Key personnel assigned to this work assignment will have training in a discipline relevant to the analyses 

contemplated under the work assignment (economics, finance, engineering, or environmental science) and 

demonstrated expertise in conducting economic impact and benefit analyses to support EPA rulemaking. 

Relevant expertise of key personnel will include, for example:

 Knowledge of the regulated industry (electric power generation and manufacturing); 

 Experience with developing analytic frameworks to quantify impacts of regulations on affected 

entities and/or with quantifying market and non-market benefits of regulations; and 

 Familiarity and demonstrated experience with implementation of a variety of analytic techniques 

(e.g., statistics, systems modeling) or software tools (spreadsheet models, relational databases). 

1.5 Role and responsibilities of project personnel

1.5.1 Project Management and Performance

As Abt Associates’ officer-level director of the EP-C-07-023 contract and overall manager for this work 

assignment, Mr. Michael Fisher, an Abt Associates Vice President and senior financial and regulatory 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 1-4

impact economist, is responsible for the overall coordination and performance of the tasks performed in 

support of the Revised 316(b) Existing Facilities Rule, including the analyses using primary and 

secondary source data. 

Ms. Tatiana Melnikova, Senior Analyst economist, is the deputy work assignment manager for the cost 

and economic impact analyses of the Revised 316(b) Existing Facilities Rule and is responsible for the 

daily activities on the project overall and data analysis efforts in particular. Ms. Melnikova has extensive 

experience developing and implementing methodologies for analyzing the cost and economic impact of 

industrial regulations pertinent to the electric power industry. She has been the lead economic impact 

analyst for 316(b) Rule analyses since 2007, and a deputy work assignment manager since 2010.

Dr. Elena Besedin, Senior Associate and senior benefits analysis economist, is the deputy work

assignment manager for the benefits analysis for the Revised 316(b) Existing Facilities rule. She will 

work closely with Mr. Ryan Stapler, Senior Analyst economist, project lead for the recreational and 

commercial fishing benefits parts of the work assignment. 

Other Abt Associates analysts and research assistants report to, and/or work with, Mr. Fisher, Ms. 

Melnikova, and Dr. Besedin in collecting data and conducting analyses. Mr. Fisher, Ms. Melnikova, Dr. 

Besedin, and Mr. Stapler are also responsible for assuring the quality of secondary data and the 

performance of analyses using these data in their respective areas of responsibility for the regulatory 

analysis. Their efforts will ensure that all data used in the project and the analyses based on these data will 

meet EPA project quality assurance requirements. 

For the implementation of the stated preference survey, as documented in more detail in Appendix 1, Dr. 

Besedin, Abt Associates’ task assignment manager for the stated preference survey, will serve as a 

technical reviewer of the work performed by Abt SRBI. Dr. Besedin will coordinate with the Abt SRBI 

work assignment manager, Mr. Jason Minser, in overseeing survey implementation and necessary quality 

assurance processes. For more details on the stated preference survey see Appendix 1.

1.5.2 Summary of Quality Assurance Project Plan Responsibilities and Implementation

In his capacity as overall work assignment manager (WAM) for the Revised 316(b) Existing Facilities 

Rule, Mr. Fisher serves as Abt Associates’ Project Quality Advisor (PQA) for work assignment 4-05 and 

will be responsible for quality assurance review. Mr. Fisher will review methodologies, data, analyses, 

and work assignment deliverables to ensure they meet Abt Associates standards for project quality, and 

also meet the standards set forth in this QAPP. In his capacity as overall PQA for the work assignment, 

Mr. Fisher will enlist PQA reviews from other personnel, as needed and appropriate, to ensure that the 

data, analyses, and work product for the various parts of work assignment 4-05 meet the quality standards 

of this QAPP. 

Each of the managers and other key personnel named above will also be responsible for implementing the 

provisions of this QAPP in their respective areas of responsibility in the work assignment. 

The Quality Assurance Manager (QAM) for the EP-C-07-023 contract, Ms. Elaine Haemisegger, is 

responsible for resolving any quality assurance concerns that arise from the technical review that cannot 

be resolved among the Abt Associates managers, and the work assignment PQA.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 2-1

2 Acquisition and Use of Primary Data

2.1 Clean Water Act Section 308 Survey of Electric Power Generating Facilities and 

Manufacturing Facilities

As described previously, EPA collected primary source data through the CWA § 308 survey of electric 

power generating facilities and manufacturing facilities for the original 316(b) Phase II and Phase III 

regulatory analyses. These data contain information on the design and operation of cooling water intake 

structures present at facilities expected to be subject to the Revised 316(b) Existing Facilities Rule. The 

questionnaire responses received in the earlier survey effort underwent substantial cleaning and quality 

validation at that time and we do not anticipate performing additional data cleaning and quality validation 

on these data for the current effort. Abt Associates will continue to use these data to support important 

elements of the analysis, particularly for the analysis of manufacturing facilities.

2.2 316(b) Stated Preference Survey

To assess the public policy significance of the ecological gains from the section 316(b) regulation for 

existing facilities, EPA received approval from the Office of Management and Budget (OMB) to conduct 

a stated preference survey. Data from the stated preference survey will be used to estimate the values 

(willingness to pay or WTP) derived by households for changes related to the reduction of fish losses at 

CWIS, and to provide information to assist in the interpretation and validation of survey responses. 

Activities associated with the survey implementation and data analysis consist of the following:

 Randomly selecting survey participants from the U.S. Postal Service DSF database 

 Printing of the survey instrument

 Mailing:

 Mailing of preview letter to notify the household that it has been selected.

 Mailing of surveys.

 Mailing of postcard reminders.

 Resending the survey to households not responding to the first survey mailing.

 Mailing the follow-up letter reminding households to complete the second survey mailing.

 Data entry and cleaning

 Conducting a follow-up study of non-respondents to the mail survey using a combination of 

telephone and priority mailing to reach non-respondents.

 Analyzing survey results. 

 Analyzing the non-response study results

EPA will be responsible for survey printing and mailing. Abt Associates’ subsidiary Abt SRBI is 

responsible for selecting survey participants from the U.S. Postal Service DSF database, conducting the 

non-response study and data entry.

1

 Abt Associates will analyze survey and non-response study results. 

 

1 Appendix 1: Data Collection and Data Processing Quality Assurance Plan, prepared by Abt SRBI, provides 

additional detail of quality assurance steps to be followed in administering the survey. 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 2-2

2.3 Selecting Survey Participants

The target population for this stated preference survey is all individuals from continental U.S. households 

who are 18 years of age or older. The population of households will be stratified into four study regions: 

Northeast, Southeast, Inland, and Pacific. The Northeast survey region includes the North Atlantic and 

Mid Atlantic 316(b) benefits regions, the Southeast survey region includes the South Atlantic and Gulf of 

Mexico 316(b) benefits regions, the Pacific region includes states on the Pacific coast, and the Inland 

region includes all non-coastal states. In addition, Abt Associates will administer a national version of the 

survey that does not require stratification.

The sample of households in each region will be randomly selected from the U.S. Postal Service Delivery 

Sequence File (DSF), which covers over 97% of residences in the United States. EPA intends to 

administer the mail survey to 7,628 households in order to achieve 2,288 completed responses assuming a 

30% completion rate for selected households. 

Each respondent will be assigned a unique survey ID to track survey responses. 

2.4 Survey Printing and Mailing

This step will be implemented by EPA in coordination with Abt Associates. Abt Associates and EPA will 

take several steps to promote response:

 Prior to mailing the survey, EPA will send the selected households a preview letter notifying 

them that they have been selected to participate in the survey and briefly describing the purpose 

of this study. 

 The mail survey will be mailed one to two weeks after the preview letter accompanied by a cover 

letter explaining the purpose of the survey. 

 All households will receive a reminder postcard approximately one week after the initial 

questionnaire mailing.

 Approximately three weeks after the first round of survey mailing, all households that have not 

responded will receive a second copy of the questionnaire with a revised cover letter. 

 A week after the second survey is mailed, a letter will be sent to remind households to complete 

the survey. 

Based on this approach to mail data collection, it is anticipated that approximately 30 percent of the 

selected households will return the completed mail survey. Since the desired number of completed 

surveys is 2,288, it will be necessary to mail surveys to 7,628 households.

The attached Appendix 1 outlines detailed quality control procedures for survey mailing and printing. 

2.5 Data Quality and Data Entry

Because the survey will be administered as a mail survey, survey responses will be scanned and entered 

into an electronic database after they are returned. Appendix 1 details quality assurance procedures for 

data entry.

After all responses have been entered, the database contents will be converted into a format suitable for 

use with a statistical analysis software package. 

All survey responses will be vetted for completeness. Additionally, respondents’ answers to the choice 

experiment questions will be tested to ensure that they are internally consistent with respect to scope and 

the neoclassical preference theory expectations of transitivity and rationality. Responses which satisfy 

transitivity (i.e., internal consistency requirements) exhibit relational relationships when separate choices 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 2-3

among policy options are compared to assess the validity of the survey response. For example, if values 

for Policy 1 are greater than Policy 2, and values for Policy 2 are greater than Policy 3, then values for 

Policy 1 should also be greater than values for Policy 3.

2.6 Data Analysis

Once the survey data have been converted into a data file, the data will be analyzed using statistical 

analysis techniques. Following standard practice, Abt Associates anticipates the estimation of a variety of 

models to assess their performance. Models to be assessed include random effects and random 

parameters (mixed) discrete choice models, now common in the stated preference literature. Abt 

Associates will assess model performance of alternative specifications using standard statistical measures 

of model fit and convergence, as detailed by Greene (2002, 2003) and Train (1998).

2.7 Conducting Non-Response Study

To determine whether there is evidence of significant non-response bias in the completed sample, Abt 

Associates will conduct a non-response follow-up study to identify potential differences in WTP 

estimates associated with respondents to the mail survey and those that did not return the questionnaire.

We will use a set of key attitudinal and socio-demographic variables that are thought to be associated with 

WTP for reducing fish mortality from cooling water intake structures to develop a short questionnaire that 

will take respondents five minutes to complete. The short questionnaire will be implemented using a dual 

frame of telephone and priority mailing.

Abt Associates will analyze the characteristics of the completed and non-completed cases from the mail 

survey and non-response questionnaire to determine whether there is any evidence of significant nonresponse bias in the completed sample. 

Appendix 1also details quality assurance procedures for telephone interviews and survey printing and 

mailing. 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 3-1

3 Acquisition and Use of Secondary Data

3.1 Sources used to identify and rationale for selection of secondary data

Abt Associates has and will identify secondary data sources for this work through research of relevant 

literature and publications, internet research, and communication with experts in the relevant fields of 

study. Abt Associates evaluates these data sources relative to the appropriateness for their specific use on 

this project. Many data sources for this work will be the same as those used to support previous 316(b) 

regulations, including data compiled and published by federal agencies.

For the cost and economic impact analysis, the federal government data sources include the Energy 

Information Administration (EIA) of the U.S. Department of Energy (DOE) (in particular, the EIA 757, 

860, 861, and 906/920/923 datasets), the Federal Energy Regulatory Commission (FERC), the U.S. Small 

Business Administration (SBA), the Bureau of Labor Statistics (BLS), the Bureau of Economic Analysis

(BEA), the U.S. Census Bureau, and the U.S. International Trade Commission (ITC). All of these data are 

compiled by the relevant federal agencies and thus meet federal government data quality standards. 

In addition, the cost and economic impact analysis will use firm financial data that have been compiled 

and reduced by private entities and information reported by publicly traded firms in Forms 10-K as 

required to be submitted by the federal Securities and Exchange Commission (SEC). These data sources 

are appropriate for this analysis based on the standards underlying their collection and publication, and 

their applicability to analyzing the cost and economic impact of the Revised 316(b) Existing Facilities 

Rule.

Abt Associates also expects to rely on analysis and outputs from the Integrated Planning Model (IPM) to 

assess the impact of regulatory requirements on electricity markets. Abt Associates and EPA used IPM in 

the previous 316(b) Phase II and Phase III regulatory analyses as well as during the proposal stage of the 

Revised 316(b) Existing Facilities Rule, to assess the total electricity market impact of regulatory 

requirements including the impact on in-scope facilities and the extended total market effect accounting 

for changes in capacity management, production levels and costs across all facilities, regardless of 

whether the facilities are directly subject to regulatory requirements. At the time of the earlier 316(b) 

Phase II regulatory analyses, Abt Associates and EPA considered a number of available analytic systems 

for supporting the 316(b) regulatory analysis and chose IPM for the 316(b) regulatory analyses based on 

the following considerations:

 Ability to receive and account for as inputs, the cost and operating effect specifications of the 

316(b) regulation;

 Ability to assess the impact of 316(b) regulatory requirements on capacity dispatch and 

utilization, capacity planning and management (i.e., capacity expansion, modifications, and 

retirements), and electricity production costs and prices;

 Level of documentation and acceptance of the models for use in assessing electricity market 

impacts of environmental regulations;

 Ability to incorporate other environmental regulatory actions in the baseline analysis;

 Ability to incorporate EPA preferences in terms of adjustments to the baseline electricity demand;

forecasts built into the model;

 Ability to meet EPA’s regulation development schedule;

 The cost of model usage.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 3-2

The currently available version of IPM (version 4.10) reflects changes in the power plant universe and 

also includes updated forecasts of electricity demand over the expected timeframe of the Revised 316(b) 

Existing Facilities regulatory compliance. In addition, the current version of IPM in its baseline also 

accounts for, recently promulgated air regulations – Clean Air Transport Rule, the Clean Air Mercury 

Rule, and the Clean Air Visibility Rule, as well as implementation of updated State Implementation 

Plans – whose impact will occur over the next five to ten years. Accordingly, the bases for selecting IPM 

to support the previous 316(b) regulatory analysis remain valid for the Revised 316(b) Existing Facilities 

regulatory analysis.

For the benefits analysis, key secondary data sources include Annual Commercial Landing Statistics

published by the National Oceanic and Atmospheric Administration (NOAA), the Marine Recreational 

Fishing Statistics Survey conducted and published by NOAA’s National Marine Fisheries Service 

(NMFS), the National Survey of Fishing, Hunting, and Wildlife-Associated Recreation published by the 

U.S. Fish and Wildlife Service (FWS) , the FWS’ Threatened and Endangered Species System (TESS) 

database, the Environmental Valuation Resource Inventory (EVRI), and other data published by EPA or 

other federal government agencies. In addition, the benefit analysis will use BEA data on economic 

indicators (e.g., GDP Deflator, Consumer Price Index) to adjust dollar values to a common dollar-year. 

These data sources are appropriate for this analysis based on their applicability (e.g., Consumer Price 

Index (CPI)) to benefits analysis of the 316(b) regulations. These data sources were the subject of 

thorough analysis of applicability to analyzing benefits of the Phase II and Phase III regulations.

Additional sources of secondary data are likely to include academic journals and other publications, 

government reports and databases, and working papers. Abt Associates staff will identify these data 

sources through literature and data searches of the EconLit database, Infotrieve’s ArticleFinder database, 

the Internet (which is frequently a good source of working papers and other documents that have not 

reached the stage of formal publication), and bibliographies or publications from government 

organizations. These data sources will be peer-reviewed, widely used and accepted in economic and 

policy analyses performed by EPA and other government and private organizations, and publicly 

available. 

3.2 Quality requirements of and acceptance criteria for secondary data

All secondary data for this work will be from original published sources or from sources of high scientific 

merit, as determined by expert judgment. As discussed above, Abt Associates selected these secondary 

data sources used for analyzing the cost and economic impacts and environmental and economic benefits 

of the Revised 316(b) Existing Facilities Rule because they are peer-reviewed, widely used and accepted 

in economic and policy analyses performed by EPA and other government and private organizations, and 

publicly available. These data sources are subject to quality assurance measures before their publication. 

As a result, the quality of these data is widely acknowledged to the extent that additional quality 

assurance is not expected. 

Abt Associates considers secondary data identified from published sources as described in Section 3.1, as 

appropriate for this effort. These data have been reviewed and approved by qualified EPA personnel and 

have been used to support cost and economic impact and environmental and economic benefit analyses to 

date. 

Going forward, as the need for additional data from secondary sources arises, the managers for this work 

assignment, Mr. Fisher, Ms. Melnikova, and Dr. Besedin, will evaluate candidate sources of this 

information for scientific strength and merit, and will ensure that selected data sources meet the standards 

set forth in this QAPP. Abt Associates will discuss using any additional secondary data with EPA before 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 3-3

incorporating them into cost and economic impact analyses and environmental and economic benefit 

analyses for this project.

3.3 Data Reduction, Validation, and Verification Procedures and Documentation

3.3.1 Data reduction procedures and documentation

Abt Associates will use data extraction and reduction procedures similar to those used in developing the 

analytical framework and datasets used in the previous rulemaking efforts. Abt Associates will work with 

the EPA WAM to determine the specific method for extracting, compiling and reducing the data wherever 

they differ from the methods used in prior efforts. Abt Associates will extract data from the original 

source, and compile, query, and categorize these data in subset databases by characteristics of interest, 

including: by state and North American Electric Reliability Corporation region, by primary energy input 

for electric power generation, or by type and regional location of waterbody from which the electric 

generating facilities withdraw cooling water.

For the 316(b) Existing Facilities Rule analyses, we anticipate that the principal changes to the secondary 

data used for the economic impact and benefit analyses will be to update datasets from the time of the 

earlier analyses to the present. Abt Associates will make any data modifications after consultation with 

the EPA WAM, and only when: (1) data are not provided in the relevant dataset, and therefore need to be 

estimated to provide a comprehensive analysis baseline; (2) data are proven to be in error and error 

reports have been submitted to the data system of origin; or (3) adjustments are necessary to ensure 

consistency of interpretation and concept within the analytical framework.

Abt Associates staff will collect full bibliographic information on all collected publications and databases, 

including date accessed for information collected from the Internet, and will save appropriate Web pages 

as PDF files. Abt Associates staff will also save any email correspondence containing EPA’s guidance 

pertinent to cost and economic impact and environmental and economic benefits analyses as PDF files. 

Abt Associates staff will document and appropriately flag in the project files and docket materials, any 

modifications to the data to differentiate between original values unchanged from the original sources and 

values estimated or modified specifically for use in the economic analyses of the Revised 316(b) Existing 

Facilities Rule and provide a date of the most recent modification/review.

3.3.2 Data validation and verification procedures and documentation

The Abt Associates WAM and PQA will perform data validation and quality assurance to ensure that all 

information submitted will meet EPA’s technical standards. Quality assurance measures include 

verification that:

 Data and other information have been appropriately and correctly transcribed from sources;

 Data used are complete, comparable, relevant, representative, reasonable given specific 

analysis/methodology;

 Information has been evaluated for scientific strength and merit;

 Information has been appropriately and correctly processed/analyzed and interpreted; 

 Technical approach, assumptions, and procedures are reasonable and logical;

 Conclusions are specific, accurate, clear, and appropriate;

 Conclusions are clearly supported by the analysis presented;

 Limitations of the data source and analyses are enumerated;

 The objective or purpose of each document is defined;

 All topics of concern specified by EPA have been addressed; and 

 Appropriate references have been provided for data sources.

 Documentation of the above is properly referenced in produced materials.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 3-4

To ensure consistency of data and verify accuracy of calculations and analysis results generated using 

these data, Abt Associates will confirm that expected relationships are met through internal review 

process. This internal review process will involve, but is not limited to, comparing summary data across 

different analyses, levels of aggregation, and between framework elements. Abt Associates WAM will 

designate a staff member, other than the one who created a given analytical file/conducted a given 

analysis, to review each analytic file. To ensure transparency of calculations/analysis and efficiency of the 

internal review process, each analytic file will have a “file information” section with the following 

information:

 Date a given file was created

 Date of most recent modification/review

 General purpose of information/analysis contained in a given file and a brief description of 

assumptions used, including data years and sources

 When appropriate, a brief description of macros, user-defined functions, and external links

 When appropriate, a brief description of information/analysis contained in each worksheet, 

including data years and sources

 Names of files/databases with data used in a file being reviewed

Each analytic Excel file will also contain a list of variable names used in a given file.

Further, to record the internal review process, each analytic file will have a Record of Internal Review/QA 

Process form to be filled out by a designated review staff member (Table 1). This form will be for 

internal Abt and EPA review purposes only, and will not be included in the public docket. The form will 

be modified, as appropriate, based on the software used and the contents/purpose of a given analytic file.

Finally, when appropriate, designated review staff will reproduce analysis results using an alternative 

methodology and/or software to ensure accuracy of analysis results. Such reviews may focus on a sample 

of data records for which analysis results will be replicated and verified. Abt Associates will investigate 

observed data irregularities and either resolve or justify them in the written documentation provided to 

EPA. All analytic files will be saved in project folders on the Abt Associates network system using a 

project folder and file management structure specified by the relevant content area manager (Mr. Fisher, 

Ms. Melnikova, or Dr. Besedin), as described at Section 3.3.3: Data storage.

All written deliverables describing data, methodology, and analysis findings will also be reviewed by a 

staff member designated by the relevant manager before being sent to EPA. Review of written work 

product will include ensuring that the most recent and correct analysis results are presented and checking 

for correct interpretation of results, clarity of discussion, proper citations and references, and spelling and 

grammar. Examples of QA procedures for written work are provided below:

 Comparing results presented in the report or memoranda with the most recent output files;

 Comparing results presented in tables and/or exhibits with the results presented in text;

 Conducting selected manual checks of analytic results presented in the report (e.g., making sure 

that individual values add up to totals);

 Assessing whether results and their interpretation are consistent with the theory, existing 

literature and prior expectation;

 Ensuring that a comprehensive list of assumptions is provided;

 Confirming that limitations and uncertainty inherent in the analysis are clearly presented;
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 3-5

 Checking references included in the report chapters against the reference list. 

When sending cost and economic impact and environmental and economic benefits analysis results to 

EPA, Abt Associates will include in each summary file a brief description of any changes to 

methodology, data, and/or assumptions that might have taken place since the previous deliverable. A 

name of each deliverable file will include a date and a deliverable version in accordance with the 

Standard Naming Convention and Version Control specifications documented under Task 2 of this work 

assignment.

Until the deliverables have gone through the internal review process discussed in this section and have 

been approved by the Abt Associates overall work assignment manager, they will contain “Internal Draft 

– Not Reviewed for QA Purposes - Deliberative, Predecisional – Do Not Quote, City, or Distribute”.

3.3.3 Data storage

As data are collected according to the foregoing discussion, Abt Associates will compile the data as 

obtained from the original sources in protected electronic folders on the Abt Associates network 

electronic storage system and/or as the original ‘hard copies’ of data reports, for data not received 

electronically. The original data are archived to preserve the data in the form in which they were 

originally received. Data compiled electronically are stored in project folders with a structure designated 

by the Abt Associates managers for the respective work assignment project areas: 

 316(b) Existing Facilities Rule cost and economic impact analysis – Ms. Tatiana Melnikova

 316(b) Existing Facilities Rule benefits analysis – Dr. Elena Besedin

Abt Associates has dedicated information technology (IT) staff responsible for the installation and 

maintenance of all hardware and software infrastructure. This includes automated daily backup of all data 

stored on networked resources. Key personnel assigned to this work assignment will work with the IT 

staff to ensure that data storage procedures are supported by the IT infrastructure.

Abt Associates applies special data management and storage procedures for data that are declared 

Confidential Business Information (CBI). These procedures prevent access to CBI by individuals who 

have not been approved for receipt and processing and/or use of CBI in accordance with the CBI

management plan developed for the overall EPA Contract No. EP-C-07-023, and as specifically 

implemented for this work assignment.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 3-6

Table 1: Record of Internal Review/Quality Assurance Process

File Name

File contains confidential business 

information (CBI) and/or information that 

could be used to reveal CBI

Yes/No

Date Reviewed

Reviewer

This file contains confidential business information (CBI) and/or information that could be used to reveal CBI.

File General

Formulas/calculations correctly reflect 

general methodology and assumptions

Summary values are consistent across 

analytical frameworks

Results have been replicated/ reproduced 

by external calculation

Results follow expected direction and 

approximate magnitude of change across 

analyzed cases (e.g., regulatory options)

References are appropriately cited

If analysis results are re-generated, 

provide information on software and 

methodology used.

Worksheet Name_1

Data are read in correctly/data sources are 

verified

Formulas/calculations correctly reflect 

general methodology and assumptions

Formulas are correctly written

Formulas refer to the appropriate cells

Summary values are consistent across 

analytical frameworks

Worksheet Name_2

Data are read in correctly/data sources are 

verified

Formulas/calculations correctly reflect 

general methodology and assumptions

Formulas are correctly written

Formulas refer to the appropriate cells

Summary values are consistent across 

analytical frameworks
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 4-1

4 Expected Work Products

Abt Associates anticipates developing various work products derived from this work, including 

memoranda on topics specified in the work plan and based on written technical direction from the EPA 

WAM; summaries of analysis findings of cost and economic impact, and environmental and economic 

benefits for alternative regulatory options under consideration; draft sections of the Cost and Economic 

Impact part and Environmental and Economic Benefits part of the NODA report; draft chapters of the 

Cost and Economic Impact TSD and Environmental and Economic Benefits Analysis TSD reports to 

support publication of the Revised 316(b) Existing Facilities Final Rule; and sections of the Federal 

Register Preambles for the Revised 316(b) Existing Facilities Final Rule. Abt Associates will include 

documentation of QAPP activities as an addendum to draft and final deliverables. 

The final step in the quality assurance process is editorial review of final deliverables. After Abt 

Associates’ internal technical reviews have been completed (and EPA WAM reviews as well), all 

comments and revisions will be addressed by the relevant project management staff – usually the 

individual content area managers – and reviewed for final acceptance by the overall project manager and 

PQA, Mr. Fisher. At this point, a skilled copy editor will conduct a final editorial review. The editor’s 

role is to verify correct grammar, spelling, and text organization/flow, as well as to ensure that the style 

and flow of the document are appropriate for the intended audience. Upon direction by the EPA WAM, 

Abt Associates will facilitate peer review of the methodologies and finished products.

If any quality assurance concerns arise from the reviews that cannot be resolved within the project team, 

Mr. Fisher will refer these concerns to the QAM for the EP-C-07-023 contract, Ms. Elaine Haemisegger, 

who is responsible for resolving any quality assurance concerns.

Subject to the time availability within a given project schedule, the respective Abt WAMs will allocate 

appropriate time for these reviews into project scheduling. The respective Abt WAMs will also dedicate 

adequate resources from the work assignment budget for technical review and quality assurance 

oversight. We will report on quality assurance activities in our monthly progress reports. All project 

deliverables will meet EPA’s standards of transparency, reproducibility, objectivity, integrity, and utility 

as specified in EPA’s Guidelines for Insuring and Maximizing the Quality, Objectivity, Utility, and 

Integrity of Information Disseminated by the Environmental Protection Agency (EPA/260R-02-008, 

October 2002), referred to as “EPA’s Information Quality Guidelines”.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Work Assignment 4-05, QAPP 5-1

5 Acceptance by the EPA WAM

We will deliver all data used in the preparation of project deliverables to the EPA WAM for review and 

approval. Upon acceptance by the EPA WAM, Abt Associates will finalize the deliverables discussed in

Section 4.
ABT SRBI

Appendix 1

Data Collection and Data Processing

Quality Assurance Plan

316(b) SURVEY

ENVIRONMENTAL PROTECTION AGENCY
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 2

ABT-SRBI DATA COLLECTION PROCEDURES AND QUALITY 

ASSURANCE

1. OVERVIEW

Abt SRBI will implement the 316(b) stated preference survey as a mail choice experiment 

questionnaire. Abt SRBI is responsible for the following steps in the survey administration 

process:

 Identifying households that will receive the mail questionnaire based on the U.S. Postal 

Service DSF database.

 Tracking survey responses on a daily basis and entering survey data.

 Conducting non-response study using a short questionnaire, which will be implemented 

using a dual frame of telephone interviews and priority mailing.

In administering the 316(b) stated preference survey, Abt SRBI will employ industry-accepted 

procedures and standards for conducting these types of survey.

Key Abt SRBI and Abt Associates personnel for the survey

Mr. Jason Minser, senior analyst, Abt SRBI project manager. Mr. Minser will be responsible for 

the overall management of the survey administration as well as response tracking and data entry. 

Mr. Minser holds an MA in Sociology and has 8 years of experience in survey design, 

management, data analysis, and reporting. Mr. Minser has designed and conducted large scale 

surveys by telephone, mail, face-to-face, Internet, and mixed mode. 

Dr. Besedin, Abt Associates’ task assignment manager for the stated preference survey, will 

serve as a technical reviewer of the work performed by Abt SRBI. Dr. Besedin will coordinate 

with the Abt SRBI work assignment manager, Mr. Minser, to ensure that Abt SRBI allocates 

appropriate time for the review into the scheduling, and dedicates adequate resources from the 

work assignment budget for technical review and quality assurance oversight. During the survey 

implementation phase, Mr. Minser will provide weekly reports to Dr. Besedin on the number of 

responses received to date, quality of the responses, and the number of responses entered in the 

electronic database. These reports will also document any quality control/quality assurance 

issues identified in the course of the work. If any issues arise, Dr. Besedin and Mr. Minser will 

identify necessary steps to correct them. 

To ensure quality of survey responses and data entry, Abt SRBI will deliver cleaned and coded 

data to Dr. Besedin when one half of the target number of completed surveys is collected. A 

description of all variables and how they are coded, as well as a memo with a description of how 

the data were cleaned, will accompany the survey response database. Dr. Besedin will review the 

initial database for completeness and consistency, and determine whether additional data 

cleaning steps are necessary. 

If any quality assurance concerns arise from the technical review that cannot be resolved 

between Dr. Besedin and Mr. Minser, they will be referred to Mr. Michael Fisher, who is the 

overall manager for the work assignment 4-05 and Abt Associates’ PQA, and has the authority to 

resolve any remaining issues.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 3

2. MAIL SURVEYS 

For this project, Abt SRBI will conduct the pilot and main studies via mail surveys. In addition,

Abt SRBI will implement a non-response study using a dual frame of telephone and priority 

mailing. This section describes the quality control (QC) procedures applicable to the pilot and 

main mail surveys as well as the priority mail non-response survey. Section 3: Telephone 

Interviews describes QC procedures that will be used in phone interviews. The mail portion of 

the survey administration will come through the Survey Operation Data Preparation (SODP) 

department. Abt SRBI’s SODP department is responsible for a variety of QC and survey support 

functions such as printing and mailing coordination, receipt control, editing, verbatim text 

coding, data entry/scanning and data cleaning. Abt SRBI designs each of these processes to 

ensure the accuracy, completeness and continuity of the data collected. Project supervisors carry 

out these activities under the guidance of the SODP management staff, who oversee the data 

preparation portion of each project’s schedule and budget. All SODP department staff possess

extensive experience in the field; many of the staff members have over 10 years of experience. 

SODP staff will perform QC at each phase of the survey process: mailings, receipt, editing, 

coding, data entry/scanning, and data cleaning. SODP staff will follow strict QC practices

tailored specifically to the survey design.

Mailing/Printing

The SODP department is capable of processing large and/or complex mailings with expertise in 

coordinating mass mailing efforts and mailings for studies that require specific materials for each 

respondent or organization. Many of these mailings include survey participation incentives or 

confidential materials that may require special handling.

For the printing for this study SODP will use mail merge technologies to generate labels for both 

the pilot and main study, for a total of 15,680 labels. In order to complete the mail merges for 

both the pilot and main study, SODP will use generated lists of addresses. SODP will assign each 

address a unique ID that clearly identifies the study region of the household, the survey version 

the household received, and a 7-digit code unique to the specific household to match later to any 

and all data received in their hard copy survey. We will send EPA labels for the pilot and main 

studies via FedEx. 

For the follow-up non-response survey printing and mailing Abt SRBI will conduct printing and 

mailing of short surveys, a mail merge generation of 500 labels, and visual checks of every 100th

survey printed and affixed with a label. Abt SRBI will complete checks to ensure surveys are 

visually presentable, correct number of pages are in the survey, and in the correct order, and 

labels are affixed to the correct surveys. Spot checking in studies of this complexity (i.e., 

multiple versions in multiple regions) are essential to avoid any misidentification and ensure 100 

percent accuracy of assignment of correct survey to version and region.

Receipt Control 

Receipt control is the process by which SODP staff keeps accurate counts of the number of 

instruments received. The SODP department will check unique IDs of the surveys and whether 

the surveys are completed, and assign batch numbers to all surveys received. An electronic file 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 4

will serve as a clearinghouse for all survey copies returned and stores vital information about 

each piece that is used to track the documents throughout the data preparation process. This 

electronic file will consist of survey id, version, completion status, date received and date 

entered, and the batch number.

Once these data are stored, they are delivered for tracking purposes to the analysts assigned to 

this study, Jason Minser. Tracking cases in this manner allows for easy identification of 

households returning completed surveys, partial surveys and non-responding households. This 

process will help us identify mailing method improvements from the pilot to the main study, 

investigate response rates in the pilot study to inform participation in the main study, and 

generate a list of non-responding households at the end of the main study to be used in the nonresponse survey.

Coding/Editing

SODP staff have extensive experience editing/coding paper instruments for the private sector as 

well as public sector studies that contain confidential materials that may require special handling.

Because the majority of this study will be conducted via mail surveys, SODP will pay additional 

attention to coding and editing of the surveys in preparation for data entry. 

Trained SODP staff will review and prepare for data entry each received and batched survey. 

The review process will include but will not be limited to confirmation of completion, 

questionnaire logic (e.g., did they answer the right questions), clarity of response, numeric code 

imputation (e.g., a number by a check box) and open end coding.

In addition to coding of mail responses, we will also code “open-end” and “other, specify” 

responses for Computer Assisted Telephone Interviewing internally using Ascribe software. 

SODP will do this for the non-response phase of the study. SODP will develop open ended 

schemas in conjunction with the analysts once 100 completed surveys are returned and data are 

entered. In this way we do not predefine codes and limit response options for survey participants. 

Once enough surveys are returned and data are entered, the analysts assigned to this study will 

assign the coding schema to be used for the rest of the surveys.

Supervisory staff will perform frequent QC checks to ensure that established editing and coding 

conventions are followed and strict security and document control measures are maintained 

throughout all the processes. 

As part of the training for this project, supervisory staff will explain editing rules for each 

question, and will read aloud and explain open-ended post-code lists. Editing rules for this 

survey will consist of how to address multiple responses in single response questions and 

extraneous markings. In this training, each editing/coding staff will independently complete the 

editing/coding of 5-10 documents. A senior coder/data coordinator will then perform a Quality 

Check to ensure that the editing instructions were correctly followed and that postcodes were 

accurately applied. The supervisory staff will thoroughly review each question to ensure that 

editing rules were correctly applied and then give immediate feedback to coding staff. The senior 

coder/data coordinator will use a QC report to document error rate.

In order to perform proper Quality Assurance of open-end postcodes, the person performing the 

Quality Check will select codes without looking at those previously applied. The coder assigned 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 5

to do the Quality Check will then check for agreement and review any discrepancies with a 

particular coding staff member to gain a better understanding of the code list. For example, a 

coder will assign codes to a set of open ended responses, followed by another coder who will 

code the same set of responses separately. The person performing the Quality Check will then 

compare the two sets of codes in order to assess if there are differences in the codes assigned to 

the same set of responses. If coding staff have difficulties understanding the post-code list, a 

senior coder will further check the list to assess whether there is a problem with the post-code list 

itself and not with the staff.

In addition to the initial check, for the first day of editing/coding, the supervisory staff will QC 

100 percent of each coder’s work as soon as possible. If this first-day review identifies problems, 

then the supervisory staff will continue to review 100 percent of the coder’s work and give 

additional feedback to the coder. If the level of QC is continuously met, the checks can be 

progressively reduced by 20 percent depending on the level of staff competency or project 

requirements, such that supervisory staff will reduce 100 percent checks to 80 percent, and so on. 

The staff supervisor will make this determination based on consistent and continuous high 

quality performance of each staff person. If errors remain frequent for a particular staff person, 

then the staff supervisor will inform SODP manager and SODP manager will remove the specific 

staff from the project. If removal is warranted, supervisory staff will quality-check all of the 

work completed by the staff before data entry.

All of these QC procedures for editing and coding will allow supervisory staff to inspect the 

work of the editing/coding staff in order to identify repeated errors, identify employees whose 

work quality is insufficient, identify inconsistencies in project design/rules, aid in the ease of 

data entry and data cleaning, minimize error, and greatly improve the overall quality of the data 

collected.

Data Entry

Once SODP completes the editing/coding of the paper surveys and the required level of QC is 

achieved, SODP supervisor will transfer the batches to the data entry stage. SODP will send each 

group of forms transferred with a transmittal coversheet identifying the batch number and total 

number of surveys. 

For the Fish and Aquatic Habitat Survey, we will use the Double Entry data entry method for 

closed-ended responses. The Double Entry method consists of data being keyed twice and 

compared. Data entry supervisor will remove discrepancies upon completion of the second entry. 

Upon review of the hard copy of each survey an operator determines which entry was correct and 

makes necessary changes. 

The Double Entry method is optimal for short field periods, high-levels of data consistency and 

protocol adherence and when data will be used for subsequent components. The Double Entry 

method allows for immediate inspection of operator entry errors (e.g., repeated offenses, 

inaccurate coding); sufficiency tracking of operator rates/quality; and checking for project 

design/rule inconsistencies. This approach is most effective in minimizing error. For any openended responses, we will use single data entry, with the text being entered verbatim.

As with the editing /coding QC measures, as soon as a data entry staff person completes entry of 

a batch of responses, a supervisor will verify/double enter the batch and provide feedback.
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 6

For the non-response portion of the study, the questionnaire will be programmed for data capture 

utilizing Teleform scanning software with a separate set up required for each of the study 

regions. Upon return through USPS, questionnaires will be sorted (separated by version/region), 

batched, and electronically receipted by ID #, batch #, and date received into an excel 

spreadsheet. Questionnaires will then be manually reviewed/edited in order to correct light fillins as well as other infrequent miscellaneous respondent error. Data entry for the non-response 

study will be accomplished utilizing scanning hardware and software. During the scan verify 

process operators will review data, enter handwritten data, and correct any errors detected by the 

software. To ensure data quality, several hard copy cases will be reviewed against data output. 

Data frequencies will then be reviewed to identify any possible errors and the data will be 

compiled/delivered in one data file per survey version.”

The general supervisor for QC of mailing, printing, receipt, coding, and data entry is Kevin 

McMahon.

Final Data Preparation

After Abt SRBI staff enter all data into the database, named WQ-4-05_T5 

_FishHabitat_Month.Date. Year, and generate data files, senior project analysts assigned to this 

study – Jason Minser– will begin the final check of all entered data using SPSS. This data 

cleaning step involves a comprehensive review of all data fields for consistency with survey 

protocols. The checks performed during this process will include but are not limited to: ensuring 

regional version variation is appropriate, identification of unallowable values, checking for nonduplicative unique ids version consistencies, incorrect implementation of skip patterns, and data 

entry errors.

Confidentiality

Abt SRBI’s SODP Center operates within a secured office environment. Entry into the space is 

possible by two methods: keyed entry and an electronic door release system. All daily traffic 

must go through the main entrance. All staff members take the utmost precautions to ensure the 

privacy of individuals responding to surveys and other modes of data collection. Data collected 

through surveys and other modes of data collection are stored on data servers which are under 

lock and key. Moreover, locked storage areas within the data preparation center safeguard 

"hardcopy" written responses and information provided by all respondents. All project and 

client-provided information is considered proprietary and is not to be shared without the 

expressed written consent of the client.

3. TELEPHONE INTERVIEWS 

To conduct the telephone interview portion of the non-response study, Abt SRBI will administer 

a short questionnaire by phone. The Abt WAM for the Stated Preference Survey task, Dr. 

Besedin, will provide the phone questionnaire. The following sections describe quality assurance 

procedures for the phone interviews. 

Interviewing Staff

The quality of the interviewing staff used on a survey is one of the most important factors 

affecting the validity, reliability and timeliness of the data collected. Abt SRBI’s telephone 
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 7

interviewing staff consists of more than 500 telephone interviewers, who are employed on a full 

or part-time basis by our firm. 

All interviewers working for Abt SRBI are thoroughly trained and closely supervised. Special 

training sessions are undertaken for each new project to help ensure Quality Control (QC) over 

the survey data collection. Abt SRBI thoroughly screens new interviewers, checks their 

references, and tests their interviewer abilities before they are hired as Abt SRBI telephone 

interviewers. All interviewers receive extensive instruction in the methods and procedures to be 

followed at Abt SRBI before they perform interviews. Supervisory staff monitor new Abt SRBI 

interviewers extremely closely during the first two weeks of their employment.

Supervisory staff monitor the performance of all our interviewers, whether they are new or have 

been with Abt SRBI for a longer period, on a regular basis twice per shift.

Employment Characteristics

Number of interviewers: 576

Training: Basic indoctrination training and study specific training

Trial period: Three shifts

Required shifts for interviewers: Minimum of three per week

Monitoring: Minimum of twice per shift

Full review: Every two weeks

Average experience: 1.8 years 

Turnover rate: Less than 10% per month

Experienced interviewers: Four years 

Spanish-speaking: Yes, more than 40 on staff

Training of Telephone Interviewers

All interviewers working for Abt SRBI receive thorough training both in general interviewing 

procedures and the protocols for a specific study. Abt SRBI interviewers follow a general 

manual on interviewing procedures developed by our operations staff. Supervisory staff may 

emphasize and recapitulate interviewing procedures for specific projects. Interviewers are trained 

in:

 An understanding of sampling procedures, generally, and the importance of adhering 

rigorously to these procedures in the field;

 An understanding of respondent selection procedures within a unit and the importance of 

following them as rigorously as the between-unit sampling procedures;

 The role of the interviewer in the survey process;
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 8

 Recommended methods for contacting potential respondents and procedures for setting 

appointments;

 Effective methods for gaining initial agreement to be interviewed;

 Methods for overcoming initial reluctance to schedule or agree to be interviewed;

 Interviewer behavior in the interview setting – how to be courteous, neutral and nonintrusive;

 How to avoid biasing responses by verbal and non-verbal cues;

 How to ask and record closed-ended questions;

 How to probe and record open-ended questions;

 How to control irrelevancies and digressions without offending the respondent; and,

 How to reassure respondents about the confidentiality of the information collected and the 

anonymity of survey respondents.

Because EPA’s non-response survey requires additional emphasis and more specialized training 

in the areas of overcoming initial reluctance to participation, interviewer behavior and recording 

of closed-ended questions, Abt SRBI will train the interviewers in these areas as well. 

In addition to focused training procedures, the field director in coordination with the project 

director will develop special training materials. Field kits will include: item-by-item interviewing 

specifications; decision algorithms for contact and location procedures; procedures to maximize 

the probability of respondent participation; proper CATI recording procedures; and additional 

reporting and quality control requirements for the study. Supervisory staff will monitor 

interviewers assigned to this project to ensure full compliance with general and emphasized 

training procedures. Abt SRBI will provide materials for this study to the EPA.

Supervisory Staff

Training does not start and stop with Abt SRBI interviewers. Abt SRBI telephone supervisors are 

instructed by field directors on specific study details, undergo additional review of interviewing 

procedures, both general and specific, and are fully involved and present in project training 

sessions.

Abt SRBI telephone surveys rely on two types of supervisors: shift supervisors and line 

supervisors or monitors. A shift supervisor works closely with the project director and the design 

team in developing project-specific field procedures. Shift supervisors are on duty each of the 

fourteen weekly shifts. Shift supervisors are responsible for meeting all production requirements. 

They oversee QC, production rates, and supervising the monitors. 

Line supervisors, or monitors, are responsible for directly overseeing individual interviewers. 

They audio-monitor the interviews being conducted, and evaluate performance of the 

interviewers in conducting the interview on question reading verbatim, pronunciation, proper 

entry of response sets and ability to build rapport with respondents.

All of the shift supervisors currently working for Abt SRBI have more than seven years of 

experience as telephone interviewers and supervisors, and are among the best in survey research.

Monitoring of Telephone Interviewers
Abt Associates Inc., November 3, 2011 Contract EP-C-07-023, Assignment 4-05, QAPP, Appendix 1 9

In this study, we will conduct a short telephone interview for main study non-respondents. A line 

monitor will silently monitor each interviewer at least twice during each interviewing shift. The 

line monitor will evaluate the interviewer on his or her performance and discusses with the shift 

supervisor any problems that an interviewer has. Before the end of the interview shift, the line 

monitor and/or shift supervisor will discuss the evaluation with the interviewer. If the interviewer 

cannot meet Abt SRBI standards, he or she will be released from work. 

All interviewers will undergo two types of monitoring. The study monitor will periodically 

review the performance of each interviewer during each interviewing shift via computer screen 

(CRT) and audio monitoring: the monitor will see on the CRT what the interviewer has recorded, 

while simultaneously hearing the interview via the audio monitor. The audio monitoring will 

allow the supervisor to determine the quality of the interviewer’s performance in terms of: (1) 

initial contact and recruitment procedures; (2) reading the questions, fully and completely, as 

written; (3) reading response categories, fully and completely, (or not reading them) according to 

study specifications; (4) whether or not open-ended questions are properly probed; (5) whether 

ambiguous or confused responses are clarified; (6) how well questions from the respondent are 

handled without alienating the respondent or biasing his/her response; (7)avoiding bias by either 

comments or vocal inflection; (8) ability to persuade wavering, disinterested or hostile 

respondents to continue the interview; and (9) general professional conduct throughout the 

interview.

The supervisor will also monitor the interviewer’s recording of survey responses on the CRT. 

The supervisor’s CRT will show the interviewer’s CRT. Consequently, the supervisor will be 

able to see whether the interviewer enters the correct code, number or verbatim in response to the 

answer. A supervisor will monitor each interviewer at least twice per shift. The proportion of 

interviews monitored for a given interviewer may be increased for a number of reasons. If a 

supervisor observes problems in an interview, he/she will intensify the interviewer’s monitoring 

to better understand the problem. If an interviewer attains a higher than average production rate 

or response rate, the supervisor will automatically intensify monitoring to be sure every 

interview is done according to specifications. The supervisor will carefully record the results of 

all study monitoring in the monitoring log for each shift.