Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP Quality Assurance Project Plan

Work Assignment No. 4

- 05, EPA Contract No. EP

- C

- 07

- 023

Assessment of the Economic Costs and Impacts and Benefits of the Final

316(b) Existing Facilities Rulemaking

Submitted by:

Abt Associates Inc. 55 Wheeler Street Cambridge, MA 02138 (617) 492

- 7100

Submitted to: Wendy Hoffman, Work Assignment Manager

and
Damon Highsmith, Contract Project Officer

U.S. Environmental Protection Agency Office of Science and Technology Engineering and Analysis Division 1200 Pennsylvania Ave., NW Washington, D.C. 20460

November 3

, 20 11Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 

### Q UALITY 

### A SSURANCE 

### P ROJECT 

### P LAN 

### (QAPP) FOR 

### : A SSESSMENT OF AND 

### P LANNING FOR THE 

### N ECESSARY 

### E CONOMICS AND 

### B ENEFITS 

### W ORK TO 

### C OMPLETE THE 

### 316( B 

### ) R ULEMAKING



## Table of Contents

1 Project Objectives, Organization, and Responsibilities

................................ ............................. 1

- 1

1.1 Project Objective/Goal

................................ ................................ ................................ ........... 1

- 1

1.2 Project requirements related to primary and secondary data

................................ .................. 1

- 2

1.3 Data analysis

................................ ................................ ................................ ........................... 1

- 3

1.4 Training and competency requirements necessary for key project personnel

........................ 1

- 3

1.5 Role and responsibil ities of project personnel

................................ ................................ ........ 1

- 3

1.5.1 Project Management and Performance

................................ ................................ ....... 1

- 3

1.5.2 Summary of Quality Assurance Project Plan Responsibilities and Implementation

................................ ................................ ................................ ........... 1

- 4

2 Acquisition and Use of Primary Data

................................ ................................ .......................... 2

- 1

2.1 Clean Water Act Section 308 Survey of Electric Power Generating Facilities and Manufacturing Facilities

................................ ................................ ................................ ....... 2

- 1

2.2 316(b) Stated Preference Survey

................................ ................................ ............................ 2

- 1

2.3 Selecting Survey Participants

................................ ................................ ................................ . 2

- 2

2.4 Survey Printing and Mailing

................................ ................................ ................................ .. 2

- 2

2.5 Data Quality and Data Entry

................................ ................................ ................................ .. 2

- 2

2.6 Data Analysis

................................ ................................ ................................ ......................... 2

- 3

2.7 Conducting Non

- Response Study

................................ ................................ ........................... 2

- 3

3 Acquisition and Use of Secondary Data

................................ ................................ ...................... 3

- 1

3.1 Sources used to identify and rationale for selection of secondary data

................................ .. 3

- 1

3.2 Quality requirements of and accept ance criteria for secondary data

................................ ...... 3

- 2

3.3 Data Reduction, Validation, and Verification Procedures and Documentation

..................... 3

- 3

3.3.1 Data reduction procedures and documentation

................................ ........................... 3

- 3

3.3.2 Data validation and verification procedures and documentation

................................ 3

- 3

3.3.3 Data storage

................................ ................................ ................................ ................ 3

- 5

4 Expected Work Products

................................ ................................ ................................ .............. 4

- 1

5 Acceptance by the EPA WAM

................................ ................................ ................................ ..... 5

- 1Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP i



### Changes to QAPP for Support of Work A ssignment 4

- 05 under Contract EP

- C

- 07

- 023

Revision 0 1

Date February 22, 2008 November 3 , 2011 September 1 6, 2011

1 : Project Objectives , Organization, and Responsibilities

1.1 : Project Objective /Goal NA Updated to reflect project objectives/goals as outlined in the work assignment 4

- 05, including a 316(b) stated preference survey. 1.2 : Project requirements related to primary and

seco ndary data NA Updated to reflect project objectives/goals as outlined in the work assignment 4

- 05, including 316(b) stated preference survey.

1.3 : Data analysis NA Non

- substantive changes to the wording. Described suppressed/withheld data estimation in more detail; provided an example. 1.4 : Training and competency requirements necessary for key project personnel NA Section added in its ent irety.

1.5 : Role and

r esponsibilities of project personnel NA Changes to reflect current project staffing/needs. Added more information on PQA’s role and QA process. Added information on interaction between Abt Associates and Abt SRBI during survey implementation. 2 : Error! Not a valid result for table.

2.3 : Selecting Survey Participants NA Section added in its entirety.

2.4 : Survey Printing and Mailing NA Section added in its entirety.

2.5 : Data Quality and Data Entry NA Section added in its entirety.

2.6 : Data Analysis NA Section added in its entirety. 2.7 : Conducting Non

Response Study NA Section added in its entirety.

3 : Acquisition and Use of Secondary Data
3.1 : Sources used to identify and rationale for selection of secondary data NA Consolidated contents of Sections 2.1 and 2.2 from the current QAPP (Revision 0)

. Updated to reflect current project needs. Added QA guidelines for accepting publications, working papers, etc. 3.2 : Quality requirements of and accep tance criteria for secondary data NA Consolidated contents of Sections 2.3 and 2.4 from the current QAPP (Revision 0)

.

3.3 : Data Reduction

, Validation , and Verification Procedures and Documentation NA Added mo re detailed information on internal review, data storage, and documentation processes. Added a list of QA measures for written work product. Made non

- substantive changes to the wording. Added Table 1

. 4 : Expected Work Products

4 : Expected Work Products NA Updated to reflect current project needs.

5 : Acceptance by the EPA WAM
5 : Acceptance by the EPA WAM NA Non

- substantive changes to the wording. Appendix 1: Abt SRBI

- Data Collection and Data Processing Quality Assurance Plan; 316(b) Survey Environmental Protection Agency Appendix 1: Abt SRBI

- Survey NA Appendix added in its entirety.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 1

- 1



### 1 Project Objectives , Organization, and Responsibilities

1.1 Project Objective /Goal

Under Contract EP

- C

- 07

- 023, work assignment 4

- 05, Abt Associates supports the Office of Science and Technology/Engineering and Analysis Division in analyzing the cost and economic impacts, environmental effe cts, and economic benefits for the Revised Existing Facilities regulation under Section 316(b) of the Clean Water Act ( referred to in this document as the Revised 316(b) Existing Facilities Rule

or the Revised 316(b) Rule )

. In doing so, Abt Associates will support the Notice of Data Ava ilability (NODA) and the final stages of rule development. Ongoing and future work in support of the Revised
316 (b) Existing Facilities Rule under this work assignment includes:

 Review and assessment of analyses undertaken and data developed and presented in support of the Revised 316 (b) Existing Facilities Proposed Rule

 P otential ly revis ing the existing methodology and/or databases

, and develop ing new analyses

 Implement ing updated/revised analyses and data sources to support the Revised 316 (b) Existing Facilities Rule

 Preparing summa ries of new cost and economic impact analyses as well as new and innovative economic benefits and aquatic health analyses , conducting the stated preference survey, analyzing and incorporating survey results in the benefits an alysis, and providing draft sections for the cost and economic impact

, and environmental and economic benefits analyses part s of the NODA

, respectively, eventually leading to the final draft version of the NODA report

 Support ing the development and assessm ent of regulatory options for the Revised 316 (b) Existing Facilities Rule to determine the three to five primary regulatory options to be presented in the final Federal Register Preamble (Preamble) and Technical Support Document (TSD) report

 Conduct ing c ost and economic impact as well as environmental and economic benefits analyses for development of Preamble, Cost and Economic Impact Analysis TSD report , and Environmental and Economic Benefits TSD report

 Support ing prepar ation of the Preamble, Final Cost and Economic Impact Analysis TSD, and Environmental and Economic Benefits TSD materials required for publication of the Revised
316 (b) Existing Facilities Final Rule
As specified in work assignment 4

- 0 5 , Abt Associates prepared and submits herewith a Quality
Assurance Project Plan (QAPP) for the data acquisition and validation, analyses , and documentation required to support the Revised 316 (b) Existing Facilities Rule . Th is QAPP contains procedures that are consistent with EPA’s q uality a ssurance guida nce specified in EPA Requirements for Quality Assurance Project Plans (EPA, 2001) . The QAPP was developed to meet the standards of accuracy and transparency for EPA regulatory analyses and will provide quality control for all primary and secondary data use d
under this w ork assignment

. Abt Associates’ quality assurance coordinators and project management team plan, implement, and assess project quality procedures throughout all project levels. These procedures are tailored to the needs of each individual tas k and will be used on this work assignment

. A copy of this QAPP will be provided to all Abt Associates staff involved in this project, including those who join the project after preparation of the QAPP.

All technical work pertinent to implementation of the 316(b) stated preference survey is performed by Abt Associates ’ survey implementation unit (Abt

- SRBI) . Accordingly, Abt

- SRBI prepared and submit s QAPP for data collection and entry performed under Task 5.1

. The plan contains proced ures that areAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 1

- 2

consistent with EPA’s quality assurance guidance and provide s quality control for all primary data collection . The QAPP for the survey implementation and data entry is provided in Attachment 1

.

1.2 Project requirements related to primary and seco ndary data

For analyzing the cost and economic impact of the Revised 316(b) Rule , Abt Associates will continue to rely on certain primary source data that were collected as part of the CWA § 308 survey of electric power generating facilities and manufactur ing facilities for the original 316(b) Phase II and Phase III regulatory analyses. These data contain information on the design and operation of cooling water intake structures present at facilities expected to be subject to the Revised 316(b) Existing Fac ilities Rule

. The questionnaire responses received in the earlier survey effort underwent substantial cleaning and quality validation at that time and we do not anticipate perform ing additional data cleaning and quality validation on these data for the current effort. Abt Associates will continue to use these data to support important elements of the analysis , particularly for the analysis of manufacturing facilities

. In addition to using this primary source data from the 316(b) facility questionnaire, Abt Associates will also rely substantially on secondary source data for this analysis. Secondary source data for this analysis include , but are not limited to :
 Data descriptive of cur rent operating and business conditions at the electric power generating and manufacturing facilities expected to be subject to the Revised 316(b) Existing Facilities Rule , and the business conditions of the firms and other entities that own these facilitie s. These data will support the assessment of baseline operations and economic/financial performance for facilities subject to regulation, and thus provide the basis for understanding the impact of the regulation on these facilities and the firms that own t hem.

 Data generally describing economic/financial conditions in, and the regulatory environment of, the elec tric power industry and manufacturing industries subject to the Revised 316(b) Existing Facilities Rule

.  Data on electricity prices, which may be a ffected by the regulation and thus in turn cause an economic impact on electricity consumers.
 Cost and/or price index data that are used to adjust dollar values from one time period to another time period.

 Information on facilities’ NPDES permitting sche dules, which is in turn used to estimate facilities’ expected compliance schedule s

. Abt Associates also expects to rely on analysis and outputs from the Integrated Planning Model (IPM) to assess the potential impact of regulatory requirements on electric power generating facilities and general market conditions within the context of regional and national electricity markets.

For analyzing the benefits expected to be achieved by the Revised 316(b) Existing Facilities Rule

, Abt Associates will rely on prima ry and secondary source data to characterize benefits of reducing impingement and entrainment. Under the previous work assignment (WA3

- 05), Abt Associates developed a stated preference survey to elicit total willingness

- to

- pay values for ecological improv ements associated with reducing impingement and entrainment of aquatic species at CWIS. Section 2 : Acquisition and Use of Primary Data provide s detail s of data collection and analysis.
Secondary source data for the environmental and economic benefits analysis include but are not limited

to :Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 1

- 3

 Data describing commercial landings and the value of commercial catch for each of the seven study regions.

 Data desc ribing recreational fishing in each of the seven study regions , including recreational catch by species, the value of recreational catch , and the number of participants

.  Data on the federally listed threatened and enda n gered species present in each analysi s region that are affected by impingement and entrainment

.  D ata on facilities’ NPDES permitting schedules, which is used to estimate facilities’ expected compliance schedul e

. Section 3 : Acquisition and Use of Secondary Data provi de s details of the secondary data sources, including criter i a for selection and use

. Section 3 also discussed the selection of IPM to support cost and economic impact analysis

.

1.3 Data analysis Abt Associates will compile, analyze, and maintain data extracted for this work in Access databases

, Excel spreadsheets , or SAS datasets and related analytic systems

. Abt Associates will conduct analyses using database queries, and will summarize data and analysis findings in databases, spreadsheets, and/ or documents. All databases , spreadsheets, and documents will cite the origin of the data and the database version (if any). We will clearly document any modifications to the data, and will app ropriately flag these modifications in the final data set to differentiate between values unchanged from the original sources and values that were estimated or modified for use in the analysis of the Revised 316(b) Existing Facilities Rule . In particular, where data have been suppressed /withheld in the or i g inal source database due to confidentiality concerns , we may need to estimate these suppressed/withheld data to ensure consistency and comprehensiveness of the modeling framework. For instance, in certain instances Economic Census data are withheld from publication because of a small number of observations in a particular economic sector and revenue range; in these instances, it may be necessary to (1) use data from adjacent revenue ranges or the total of revenue ran ges to describe characteristics of the missing sector/revenue range combination, or (2) use data from a higher level of sector definition (e.g., NAICS 5

- digit sector instead of
6

- digit sector) to describe characteristics of the missing data item.

1.4 Training and competency requirements necessary for key project personnel Key personnel assigned to this work assignment will have training in a discipline relevant to the analyses contemplated under th e w ork assignment (economics, finance, engineering, or environm ental science) and demonstrated expertise in conducting economic impact and benefit analyses to support EPA rulemaking. Relevant expertise of key personnel will include, for example:

 Knowledge of the regulated industry (electric power generation and manufa cturing);

 Experience with developing analytic frameworks to quanti f y impacts of regulations on affected entities and/or with quantifying market and non

- market benefits of regulations; and

 Familiarity and demonstrated experience with implementation of a variety of analytic techniques (e.g., statistics, systems modeling) or software tools (spreadsheet models, relational databases).

1.5 Role and r esponsibilities of project personnel

1.5.1 Project Manag ement and Performance
As Abt Associates’ officer

- level director of the EP

- C

- 07

- 023 contract and overall manager for this work assignment , Mr. Michael Fisher , an Abt Associates Vice President and senior financial and regulatoryAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 1

- 4

impact economist, is responsi ble for the overall coordination and performance of the tasks performed in support of the Revised 316(b) Existing Facilities Rule

, including the analyse s using primary and secondary source data.

Ms. Tatiana Melnikova , Senior Analyst economist, is the depu ty work assignment manager for the cost and economic impact analyse s of the Revised 316 (b) Existing Facilities Rule and is responsible for the daily activities on the project overall and data analysis efforts in particular

. Ms. Melnikova has extensive

expe rience developing and implementing methodologies for analyzing the cost and economic impact of industrial regulations pertinent to the electric power industry. She has been the lead economic impact analyst for 316(b) Rule analyses since 2007

, and a deputy work assignment manager since 2010.

Dr. Elena Besedin , Senior Associate and senior benefits analysis economist, is the deputy work assignment manager for the benefits analysis for the Revised 316 (b) Existing Facilities rule

. She will work closely with Mr. Ryan Stapler, Senior Analyst economist, project lead for the recreational and commercial fishing benefits parts of the work assignment

. Other Abt Associates analysts and research assistants report to, and/ or work with, Mr. Fisher

, Ms. Melnikova, and Dr. Besedin in collecting data and conducting analyses. Mr. Fisher , Ms. Melnikova, Dr. Besedin , and Mr. Stapler are also responsible for assuring the quality of secondary data and the performance of analyses using these data in their respective areas o f responsibility for the regulatory analysis. Their efforts will ensure that all data used in the project and the analyses based on these data will meet EPA project quality assurance requirements.
For the implementation of the stated preference survey, as documented in more detail in Appendix 1, Dr. Besedin, Abt Associates’ task assignment manager for the stated preference survey, will serve as a technical reviewer of the work performed by Abt SRBI. Dr. Besedin will coordinate with the Abt SRBI work assign ment manager, M r

. Jason Minser , in overseeing survey implementation and necessary quality assurance processes. For more details on the stated preference survey see 

### Appendix 1

. 1.5.2 Summary of Quality Assurance Project Plan Responsibilities and Implementation

In his capacity as overall w ork assignment manager (WAM) for the Revised 316(b) Existing Facilities Rule, Mr. Fisher serves as Abt Associates’ Project Quality Advisor (PQA) for work assignment 4

- 05 and will be responsible for quality assurance review. Mr. Fisher will review methodologies, data, analyses, and work assignment deliverables to ensure they meet Abt Associates standards for project quality, and also meet the standards set forth in this Q APP

. In his capacity as overall PQA for the work assignm ent, Mr. Fisher will enlist PQA reviews from other personnel, as needed and appropriate, to ensure that the data, analyses, and work product for the various parts of work assignment 4

- 05 meet the quality standards of this QAPP.

Each of the managers and ot her key personnel named above will also be responsible for implementing the provisions of this QAPP in their respective areas of responsibility in the work assignment

. The Quality Assurance Manager (QAM) for the EP

- C

- 07

- 023 contract, Ms. Elaine Haemisegge r , is responsible for resolving any quality assurance concerns that arise from the technical review that cannot be resolved among the Abt Associates managers, and the work assignment PQA.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 2

- 1



### 2 Acquisition and Use of Primary Data

2.1 Clean Wa ter Act Section 308 Survey of Electric Power Generating Facilities and Manufacturing Facilities As described previously, EPA collected primary source data through the CWA § 308 survey of electric power generating facilities and manufacturing facilities for the original 316(b) Phase II and Phase III regulatory analyses. These data contain information on the design and operation of cooling water intake structures present at facilities expected to be subject to the Revised 316(b) Existing Facilities Rule. The questionnaire responses received in the earlier survey effort underwent substantial cleaning and quality validation at that time and we do not anticipate performing additional data cleaning and quality validation on these data for the current effort. Abt A ssociates will continue to use these data to support important elements of the analysis, particularly for the analysis of manufacturing facilities.

2.2 316(b) Stated Preference Survey To assess the public policy significance of the ecological gains from the section 316(b) regulation for existing facilities, EPA received approval from the Office of Management and Budget (OMB) to conduct a stated preference survey. Data from the stated p reference survey will be used to estimate the values (willingness to pay or WTP) derived by households for changes related to the reduction of fish losses at CWIS, and to provide information to assist in the interpretation and validation of survey response s. Activities associated with the survey implementation and data analysis consist of the following:

 Randomly selecting survey participants from the U.S. Postal Service DSF database

 Printing of the survey instrument

 Mailing :

 Mailing of preview letter to notify the household that it has been selected.

 Mailing of surveys.

 Mailing of postcard reminders.

 Resending the survey to households not responding to the first survey mailing.

 Mailing the follow

- up letter reminding households to complete the second surve y mailing.

 Data entry and cleaning

 Conducting a follow

- up study of non

- respondents to the mail survey using a combination of telephone and priority mailing to reach non

- respondents.

 Analyzing survey results.

 Analyzing the non

- response study results

EPA w ill be responsible for survey printing and mailing. Abt Associates ’ subsidiary Abt SRBI is responsible for selecting survey participants from the U.S. Postal Service DSF database, conducting the

non

- response study and data entry

. 1 Abt Associates will anal yze survey and non

- response study results

.

1 Appendix 1: Data Collection and Data Processing Quality Assurance Plan , prepared by Abt SRBI, provides additional detail of quality assurance steps to be followed in administering the survey.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 2

- 2

2.3 Selecting Survey Participants The target population for this stated preference survey is all individuals from continental U.S. households who are 18 years of age or older. The population of households will be st ratified into four study regions: Northeast, Southeast, Inland, and Pacific. The Northeast survey region includes the North Atlantic and Mid Atlantic 316(b) benefits regions, the Southeast survey region includes the South Atlantic and Gulf of Mexico 316(b) benefits regions, the Pacific region includes states on the Pacific coast, and the Inland region includes all non

- coastal states. In addition, Abt Associates will administer a national version of the survey that does not require stratification.
The sampl e of households in each region will be randomly selected from the U.S. Postal Service Delivery Sequence File (DSF), which covers ov er 97% of residences in the United States. EPA intends to administer the mail survey to 7,628 households in order to achieve 2,288 completed responses assuming a 30% completion rate for selected households.
Each respondent will be assigned a unique survey ID to track survey responses

.

2.4 Survey Printing and Mailing This step will be implemented by EPA in coordination with Abt As sociates. Abt Associates and EPA will take several steps to promote response:

 Prior to mailing the survey, EPA will send the selected households a preview letter notifying them that they have been selected to participate in the survey and briefly describing the purpose of this study.

 The mail survey will be mailed one to two weeks after the preview letter accompanied by a cover letter explaining the purpose of the survey.

 All households will receive a reminder postcard approximately one week after the initial questionnaire mailing.

 Approximately three weeks after the first round o f survey mailing, all households that have not responded will receive a second copy of the questionnaire with a revised cover letter.

 A week after the second survey is mailed, a letter will be sent to remind households to complete the survey.

Based on this approach to mail data collection, it is anticipated that approximately 30 percent of the selected households will return the completed mail survey. Since the desired number of completed surveys is 2,288, it will be necessary to mail surveys to 7,628 households.

The attached Appendix 1 outlines detailed quality control procedures for survey mailing and printing

.

2.5 Data Quality and Data Entry

Because the survey will be administered as a mail survey, survey responses will be scanned and entered
into an el ectronic database after they are returned

. Appendix 1 details quality assurance procedures for data entry.
After all responses have been entered, the database contents will be converted into a format suitable for use with a statistical analysis software p ackage

. All survey responses wi ll be vetted for completeness. Additionally, respondents’ answers to the choice experiment questions will be tested to ensure that they are internally consistent with respect to scope and the neoclassical preference theory expectations of transitivity and rationality . Responses which satisfy transitivity (i.e., internal consistency requirements) exhibit relational relationships when separate choicesAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 2

- 3

among policy options are compared to assess the validity of the survey res ponse . For example, if values for Policy 1 are greater than Policy 2, and values for Policy 2 are greater than Policy 3, then values for
P olicy 1 should also be greater than values for Policy 3.

2.6 Data Analysis Once the survey data have been converted into a data file, the data will be analyzed using statistical analysis techniques . Following standard practice, Abt Associates anticipates the estimation of a variety of models to assess their performance. Models to be assessed include random effects and rand om parameters (mixed) discrete choice models, now common in the stated preference literature. Abt Associates will assess model performance of alternative specifications using standard statistical measures of model fit and convergence, as detailed by Green e (2002, 2003) and Train (1998).

2.7 Conducting Non

- Response Study To determine whether there is evidence of significant non

- response bias in the completed sample, Abt Associates will conduct a non

- response follow

- up study to identify potential differences in WTP estimates associated with respondents to the mail survey and those that did not return the questionnaire.

We will use a set of key attitudinal and socio

- demographic variables that are thought to be associated with WTP for reducing fish mortality from c ooling water intake structures to develop a short questionnaire that will take respondents five minutes to complete. The short questionnaire will be implemented using a dual frame of telephone and priority mailing.
Abt Associates will analyze the character istics of the completed and non

- completed cases from the mail survey and non

- response questionnaire to determine whether there is any evidence of significant non

response bias in the completed sample.
Appendix 1 also details quality assurance procedures f or telephone interviews and survey printing and mailing.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 3

- 1



### 3 Acquisition and Use of Secondary Data

3.1 Sources used to identify and rationale for selection of secondary data

Abt Associates has and will identif y secondary data sources for th is work through research of relevant literature and publications, internet research, and communication with experts in the relevant fields of study. Abt Associates evaluate s these data sources relative to the appropriateness for their specific use on this project. Many data sources for this work will be the same as those used to support previous 316(b) regulations

, including data compiled and published by federal agencies

. For the cost and economic impact analysis, t he federal government data sources inclu de the Energy Information Administration (EIA) of the U.S. Department of Energy (DOE) (in particular, the EIA 757, 860, 861, and 906 /920/923 datasets), the Federal Energy Regulatory Commission (FERC) , the U.S. Small Business Administration (SBA) , the Bureau of Labor Statistics (BLS) , the Bureau of Economic Analysis (BEA)

, the U.S. Census Bureau, and the U.S. International Trade Commission (ITC) . All of the se data are compiled by the relevant federal agencies and thus meet federal government data qualit y standards.

In addition, the cost and economic impact analysis will use firm financial data that have been compiled and reduced by private entities and information reported by publicly traded firms in Forms 10

- K as required to be submitted by the federal Securities and Exchange Commission (SEC)

. These data sources are appropriate for this analysis based on the standards underlying their collection and publication, and their applicability to analyzing the cost and economic impact of the Revised 316 (b) Exis ting Facilities Rule

. Abt Associates also expects to rely on analysis and outputs from the Integrated Planning Model (IPM) to assess the impact of regulatory requirements on electricity markets. Abt Associates and EPA used IPM in the previous 316(b) Phase II and Phase III regulatory analyses as well as during the proposal stage of the Revised 316(b) Existing Facilities Rule

, to assess the total electricity market impact of regulatory requirements including the impact on in

- scope facilities and the extended total market effect accounting for changes in capacity management, production levels and costs across all facilities, regardless of whether the facilities are directly subject to regulatory requirements. At the time of the earlier 316(b) Phase II regulator y analyses, Abt Associates and EPA considered a number of available analytic systems for supporting the 316(b) regulatory analysis and chose IPM for the 316(b) regulatory analyses based on the following considerations:

 Ability to receive and account for as inputs, the cost and operating effect specifications of the 316(b) regulation ;
 Ability to assess the impact of 316(b) regulatory requirements on capacity dispatch and utilization, capacity planning and management (i.e., capacity expansion, modifications, and retirements), and electricity production costs and prices ;
 Level of documentation and acceptance of the models for use in assessing electricity market impacts of environmental regulations ;
 Ability to incorporate other environmental regulatory actions in the baseline analysis ;
 Ability to incorporate EPA preferences in terms of adjustments to the baseline electricity demand ; forecasts built into the model ;
 Ability to meet EPA’s regulation development schedule ;
 The cost of model usage.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 3

- 2

The currently avail able version of IPM (version 4.10) reflects changes in the power plant universe and also includes updated forecasts of electricity demand over the expected timeframe of the Revised 316(b) Existing Facilities regulatory compliance. In addition, the current version of IPM in its baseline also accounts for, recently promulgated air regulations

– Clean Air Transport Rule, the Clean Air Mercury Rule, and the Clean Air Visibility Rule, as well as implementation of updated State Implementation Plans

– whose impact will occur over the next five to ten years. Accordingly, the bases for selecting IPM to support the previous 316(b) regulatory analysis remain valid for the Revised 316(b) Existing Facilities regulatory analysis.
For the benefits analysis, key secondary d ata sources include Annual Commercial Landing Statistics published by the National Oceanic and Atmospheric Administration ( NOAA ) , the Marine Recreational Fishing Statistics Survey conducted and published by NOAA’s National Marine Fisheries Service (NMFS)

, the National Survey of Fishing, Hunting, and Wildlife

- Associated Recreation published by the U.S. Fish and Wildlife Service (FWS)

, the FWS’ Threatened and Endangered Species System (TESS) database, the Environmental Valuation Resource Inventory (EVRI), and other data published by EPA or other federal government agencies. In addition, the benefit analysis will use BEA data on economic indicators (e.g., GDP Deflator, Consumer Price Index) to adjust dollar values to a common dollar

- year. These data sources are appropriate for this analysis based on their applicability (e.g., Consumer Price Index (CPI) ) to benefits analysis of the 316(b) regulations. These data sources were the subject of thorough analysis of applicability to analyzing benefits of the Phase II and Phase III regulations.
Additional sources of secondary data are likely to include academic journals and other publications, government reports and databases, and working papers. Abt Associates staff will identify these data sources through literatur e and data search es of the EconLit database, Infotrieve’s ArticleFinder database, the Internet (which is frequently a good source of working papers and other documents that have not reached the stage of formal publication), and bibliographies or publicatio ns from government organizations. These data sources will be peer

- reviewed, widely used and accepted in economic and policy analyses performed by EPA and other government and private organizations, and publicly available.

3.2 Quality requirements of and accep tance criteria for secondary data All secondary data for this work will be from original published sources or from sources of high scientific merit, as determined by expert judgment. As discussed above, Abt Associates selected these secondary data sources used for analyzing the cost and economic impacts and environmental and economic benefits of the Revised 316(b) Existing Facilities Rule because they are peer

- reviewed, widely used and accepted in economic and policy analyses performed by EPA and other gov ernment and private organizations, and publicly available. These data sources are subject to quality assurance measures before their publication. As a result, the quality of these data is widely acknowledged to the extent that additional quality

assurance is not expected.

Abt Associates consider s secondary data identified from published sources as described in Section 3.1 , as appropriate for this ef

fort. These data have been reviewed and approved by qualified EPA personnel and have been used to support cost and economic impact and environmental and economic benefit analyses to date

. Going forward, as the need for additional data from secondary sourc es arises, the managers for this work assignment

, Mr. Fisher, Ms. Melnikova, and Dr. Besedin

, will evaluate candidate sources of this information for scientific strength and merit

, and will ensure that selected data sources meet the standards set forth in this QAPP. Abt Associates will discuss using any additional secondary data with EPA beforeAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 3

- 3

incorporating them into cost and economic impact analyses and environmental and economic benefit analyses for this project.

3.3 Data Reduction
, Validation , and Verification Procedures and Documentation

3.3.1 Data reduction procedures and documentation Abt Associates will use data extraction and reduction procedures similar to those used in developing th e analytical framewor k and data sets used in the previous rule making effort s

. Abt Associates will work with the EPA WAM to determine the specific method for extracting, compiling and reducing the data wherever they differ from the methods used in prior effort s

. Abt Associates w ill extract data from the original

source, and compile, query, and categorize these data in subset databases by characteristics of interest

, including: by state and North American Electric Reliability Corporation region, by primary energy input for electri c power generation, or by type and regional location of waterbody from which the electric generating fa cilities withdraw cooling water

. For the 316(b) Existing Facilities Rule analyses, we anticipate that the principal changes to the secondary data used fo r the economic impact and benefit analyses will be to update datasets from the time of the earlier analyses to the present

. Abt Associates will make any data modifications after consultation with the EPA WAM, and only when: (1) data are not provided in the relevant dataset , and therefore need to be estimated to provide a comprehensive analysis baseline ; (2) data are proven to be i n error and error reports have been submitted to the data system of origin ; or (3) adjustments are necessary to ensure consistenc y of interpretation and concept within the analytical framework

. Abt Associates staff will collect full bibliographic information on all collected publications and databases

, including date accessed f or information collected from the Internet, and will sa ve appropriate Web pages as PDF files. Abt Associates staff will also save any email correspondence containing EPA’s guidance pertinent to cost and economic impact and environmental and economic benefits analyses as PDF files. Abt Associates staff will doc ument and appropriately flag in the project files and docket materials, any modifications to the data to differentiate between original values unchanged from the original sources and values estimated or modified specifically for use in the economic analyse s of the Revised 316 (b) Existing Facilities Rule and provide a date of the most recent modification/review

.
3.3.2 Data validation and verification procedures and documentation
The Abt Associates WAM and PQA will perform data validation and quality assurance to ensure that all information submitted will meet EPA’s technical standards. Quality assurance measures include verification that:  Data and other information have been appropriately and correctly transcribed from sources;  Data used are complete, comparable, relevant, representative, reasonable given specific analysis/methodology ;  Information has been evaluated for scientific strength and merit;  Information has been appropriately and correctly processed/analyzed and interpre ted;  Technical approach, assumptions, and procedures are reasonable and logical ;  Conclusions are specific, accurate, clear, and appropriate;  Conclusions are clearly supported by the analysis presented;  Limitations of the data source and analyses are enumerated;  The objective or purpose of each document is defined;  All topics of concern specified by EPA have been addressed; and  Appropriate references have been provided for data sources.  Documentation of the above is properly referenced in produced mat erials.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 3

- 4

To ensure consistency of data and verify accuracy of calculations and analysis results generated using these data , Abt Associates will confirm that expected relat ionships are met through internal review

process . This internal review process will involve , but is not limited to, comparing summary data across different analyses, levels of aggregation

, and between framework elements. Abt Associates WAM will designate a staff member

, other than the one who created a given analytical file/conducted a gi ven analysis

, to review e ach analytic file. To ensure transparency of calculations/analysis and efficiency of the internal review process, each analytic file will have a “file information” section with the following information :
 Date a given file was creat ed

 Date of most recent modification /review

 General purpose of information/analysis contained in a given file and a brief description of assumptions used , including data year s and source s
 When appropriate, a brief d escri ption of macros , user

- defined functions , and external links

 When appropriate, a brief description of information/analysis contained in each worksheet

, including data year s and source s
 Names of files /databases with data used in a file being reviewed
Each analytic Excel fi le will also contain a list of variable names used in a given file.

Further, t o record th e internal review process, each analytic file will have a Record of Internal R eview/QA
Process form to be filled out by a designated review staff member ( Table 1 ). This form will be for internal Abt and EPA review purposes only, and will not be included in the public docket. Th e form will be modified , a s appropriate, based on the software used and the c ontents/purpose of a given analytic file. Finally, when appropriate, designated review staff will reproduce analysis results using an alternative methodology and/or software to ensure accuracy of analysis results. Such reviews may focus on a sample of data records for which analysis results will be replicated and verified. Abt Associates will investigate observed data irregularities and either resolve or justify them in the written doc umentation provided to
EPA. All analytic files will be saved in project folder s on the Abt Associates network system using a project folder and file management structure specified by the relevant content area manager (Mr. Fisher, Ms. Melnikova, or Dr. Bese din), as described at Section 3.3.3 : Data storage

. All written deliverables describing data, methodology, and analysis findings will also be reviewed by a staff member designate d by the relevant manager before being sent to EPA. Review of written work product will include ensuring that the most recent and correct analysis results are presented and checking for correct interpretation of results, clarity of discussion, proper citat ions and references, and spelling and

grammar. Examples of QA procedures for written work are provided below:

 Comparing results presented in the report or memoranda with the most recent output files ;
 Comparing results presented in tables and/or exhibits wi th the results presented in text ;
 Conducting selected manual checks of analytic results presented in the report (e.g., making sure that individual values add up to totals) ;
 A ssess ing whether results and their interpretation are consistent with the theory, existing literature and prior expectation ;
 Ensuring that a comprehensive list of assumptions is provided ;
 Confirming that limitations and uncertainty inherent in the analysis are c learly presented ;Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 3

- 5

 C heck ing references included in the report chapter s against the reference list.
When sending cost and economic impact and environmental and economic benefits analysis results to EPA, Abt Associates will include in each summary file a brief description of any changes to methodology, data, and/or assumptions that might have taken place since the previous deliverable. A name of each deliverable file will include a date and a deliverable version in accordance with the Standard Naming Conve ntion and Version Control specifications documented under Task 2 of this work assignment

. Until the deliverables have gone through the internal review process discussed in this section and have been approved by the Abt Associates overall work assignment ma nager , they will contain “ Internal Draft

– Not Reviewed for QA Purposes

- Deliberative, Predecisional

– Do Not Quote, City, or Distribute ”

. 3.3.3 Data storage As data are collected according to the foregoing discussion, Abt Associates will compile the data as obtained from the original sources in protected electronic folders on the Abt Associates network electronic storage system and/or as the original ‘hard copies’ of data reports, for data not received electronically. The original data are archiv ed to preserve the data in the form in which they we re originally received. Data compiled electronically are stored in project folders with a structure designated by the Abt Associates managers for the respective work assignment project areas:

 316(b) Exis ting Facilities Rule cost and economic impact analysis

– Ms. Tatiana Melnikova  316(b) Existing Facilities Rule benefits analysis

– Dr. Elena Besedin
Abt Associates has dedicated information technology (IT) staff responsible for the installation and mainten ance of all hardware and software infrastructure. This includes automated daily backup of all data stored on networked resources. Key personnel assigned to this work assignment will work with the IT staff to ensure that data storage procedures are supporte d by the IT infrastructure.
Abt Associates applies special data management and storage procedures for data that are declared Confidential Business Information (CBI). These procedures prevent access to CBI by individuals who have not been approved for recei pt and processing and/or use of CBI in accordance with the C BI management plan developed for the overall EPA Contract No. EP

- C

- 07

- 023

, and as specifically implemented for this work assignment

.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 3

- 6

Table 1 : Record of Internal Revie w/ Quality A ssurance Process

File Name

File contains confidential business information (CBI) and/or information that could be used to reveal CBI Yes/No

Date Reviewed

Reviewer

This file contains confidential business information (CBI) and/or information that could be used to reveal CBI.

File General
Formulas/calculations correctly reflect general methodology and assumptions

Summary values are consistent across analytical frameworks

Results have been replicated/ reproduced by external calculation

Results follow expected direction and approximate magnitude of change across analyzed cases (e.g., regulatory options)

References are appropriately cited

If analysis results are re

- generated, provide information on software and methodology used. Worksheet Name_1
Data are read in correctly /data sources are verified

Formulas /calculations correctly reflect general methodology and as s u mptions

Formulas are correctly written

Formulas refer to the appropriate cells

Summary values are consistent across analytical frameworks Worksheet Name_2
Data are read in correctly/data sources are verified

Formulas/calculations correctly reflect general methodology and assumptions

Formulas are correctly written

Formulas refer to the appropriate cells

Summary values are consistent across analytical frameworksAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 4

- 1



### 4 Expected Work Products Abt Associates anticipates developing various work products derived from this work , including memoranda on topics specified in the work plan and based on written technical direction from the EPA WAM; summaries of analysis findings of cost and economic impact

, and environmental and economic benefits for alternative regulatory options under considera tion; draft sections of the Cost and Economic Impact part and Environmental and Economic Benefits part of the NODA report; draft chapters of the Cost and Economic Impact TSD and Environmental and Economic Benefits Analysis TSD report s to support publicatio n of the Revised 316 (b) Existing Facilities Final Rule; and sections of the Federal Register P reamble s for the Revised 316(b) Existing Facilities Final Rule

. Abt Associates will include documentation of QAPP activities as an addendum to draft and final deliverables.
The final step in the quality assurance process is editorial review of final deliverables. After Abt Associates’ internal technical reviews have b een completed (and EPA WAM reviews as well), all comments and revisions will be addressed by the relevant project management staff

– usually the individual content area managers

– and reviewed for f inal acceptance by the overall p roject m anager and PQA , M r. Fisher. At this point, a skilled copy editor will conduct a final editorial review. The editor’s role is to verify correct grammar, spelling, and text organization/flow, as well as to ensure that the style and flow of the document are appropriate for th e intended audience. Upon direction by the EPA WAM, Abt Associates will facilitate peer review of the methodologies and finished products.
If any quality assurance concerns arise from the review s that cannot be resolved within the project team,
Mr. Fisher will refer these concerns to the QAM for the EP

- C

- 07

- 023 contract, Ms. Elaine Haemisegger, who is responsible for resolving any quality assurance concerns.
Subject to the time availability within a given project schedule, t he respective Abt WAMs will allocate appropriate time for the se review s into project scheduling. The respective Abt WAMs will also dedicate adequate resources from the work assignment budget for technical review and quality assurance oversight. We will report on quality assurance activities in our monthly progress reports. All project deliverables will meet EPA’s standards of transparency, reproducibility, objectivity, integrity, and utility as specified in EPA’s Guidelines for I nsuring and Maximizing the Quality, Objecti vity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency (EPA/260R

- 02

- 008, October 2002), referred to as “EPA’s Information Quality Guidelines ”

.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Work Assignment 4

- 05, QAPP 5

- 1



### 5 Acceptance by the EPA WAM

We will deliver all dat a used in the preparation of project deliverables to the EPA WAM for review and approval. Upon acceptance by the EPA WAM, Ab t Associates will finalize the deliverables discussed in
Section 4

.

## ABT SRBI
Appendix 1 Data Collection and Data Processing Quality Assurance Plan

316(b) SURVEY ENVIRONMENTAL PROTECTION AGENCYAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 2



## ABT

- SRBI DATA COLLECTION PROCEDURES AND QUALITY ASSURANCE

1. OVERVIEW

Abt SRBI will implement t he 316(b) stated preference survey as a mail choice experiment questionnaire. Abt SRBI is responsible for the following steps in the survey administration
process:  Identifying households that will receive the mail questionnaire based on the U.S. Postal Ser vice DSF database.  Tracking survey responses on a daily basis and entering survey data.  Conducting non

- response study using a short questionnaire , which will be implemented using a dual frame of telephone interviews and priority mailing.

In administering the 316(b) stated preference survey, Abt SRBI will employ industry

- accepted procedures and standards for conducting these types of survey.

Key Abt SRBI and Abt Associates personnel for the survey

Mr. Jason Minser, senior analyst, Abt SRBI project manager. Mr. Minser will be responsible for the overall management of the survey administration as well as response tracking and data entry. Mr. Minser holds an MA in Sociology and has 8 years of experience in survey design, management, data analysis, and reporting. Mr. Minser has designed and conducted large scale surveys by telephone, mail, face

- to

- face, Internet, and mixed mode.

Dr. Besedin, Abt Associates’ task assignment manager for the stated preference survey, will serve as a technica l reviewer of the work performed by Abt SRBI. Dr. Besedin will coordinate with the Abt SRBI work assignment manager, Mr. Minser, to ensure that Abt SRBI allocates appropriate time for the review into the scheduling, and dedicates adequate resources from th e work assignment budget for technical review and quality assurance oversight. During the survey implementation phase, Mr. Minser will provide weekly reports to Dr. Besedin on the number of responses received to date, quality of the responses, and the numb er of responses entered in the electronic database. These reports will also document any quality control/quality assurance issues identified in the course of the work. If any issues arise, Dr. Besedin and Mr. Minser will identify necessary steps to correct them.

To ensure quality of survey responses and data entry, Abt SRBI will deliver cleaned and coded data to Dr. Besedin when one half of the target number of completed surveys is collected. A description of all variables and how they are coded, as well as a memo with a description of how the data were cleaned, will accompany the survey response database. Dr. Besedin will review the initial database for completeness and consistency, and determine whether additional data cleaning steps are necessary.

If any quality assurance concerns arise from the technical review that cannot be resolved between Dr. Besedin and Mr. Minser, they will be referred to Mr. Michael Fisher, who is the overall manager for the work assignment 4

- 05 and Abt Associates’ PQA, and ha s the authority to resolve any remaining issues.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 3



### 2. MAIL SURVEYS

For this project, Abt SRBI will conduct the pilot and main studies via mail surveys . In addition, Abt SRBI will implement a non

- response study using a dual frame of telephone and priority mai ling. This section describes the quality control (QC) procedures applicable to the pilot and main mail surveys as well as the priority mail non

- response survey. Section 3 : Telephone Interviews describes QC procedures that will be used in phone interviews. The mail portion of the survey administration will come through the Survey Operation Data Preparation (SODP) department. Abt SRBI’s SODP department is responsible for a variety of QC and survey support functions such as printing and mailing coordination, receipt control, editing, verbatim text coding, data entry/scanning and data cleaning. Abt SRBI designs e ach of these processes to

ensure the accuracy, completeness and continuity of the data collected. Project supervisors carry

out t hese activities under the guidance of the SODP m anagement staff, who oversee the data preparation portion of each project’s schedule and budget. All SODP depart ment staff possess extensive experience in the field ; many of the staff members have over 10 years of experience.

SODP staff will perform QC at each phase of the survey process: mailings, receipt, editing, coding, data entry/scanning

, and data cleaning

. SODP staff will follow strict QC practices tailored specifically to the survey design.

Mailing/Printing

The SODP d epartment is capable of processing large and/or complex mailings with expertise in coordinating mass mailing efforts and mailings for studies that require specific materials for each respondent or organization. Many of these mailings include survey partici pation incentives or confidential materials that may require special handling.

For the printing for this study SODP will use mail merge technologies to generate labels for both the pilot and main study, for a total of 15,680 labels. In order to complete t he mail merges for both the pilot and main study, SODP will use generated lists of addresses. SODP will assign each address a unique ID that clearly identifies the study region of the household, the survey version the household received, and a 7

- digit code unique to the specific household to match later to any and all data received in their hard copy survey. We will send EPA labels for the pilot and main studies via FedEx.

For the follow

- up non

- response survey printing and mailing Abt SRBI will conduct pr inting and mailing of short surveys, a mail merge generation of 500 labels, and visual checks of every 100 th 

### survey printed and affixed with a label. Abt SRBI will complete checks to ensure surveys are visually presentable, correct number of pages are in t he survey, and in the correct order, and labels are affixed to the correct surveys. Spot checking in studies of this complexity (i.e., multiple versions in multiple regions) are essential to avoid any misidentification and ensure 100 percent accuracy of as signment of correct survey to version and region.

Receipt Control

Receipt control is the process by which SODP s taff keep s accurate counts of the number of instruments received. The SODP department will check unique IDs of the surveys and whether the s urveys are completed, and assign batch numbers to all surveys received. An electronic fileAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 4



### will serve as a clearinghouse for all survey copies returned and stores vital information about each piece that is used to track the documents throughout the data pr eparation process. This electronic file will consist of survey id, version, completion status, date received and date entered, and the batch number.

Once these data are stored, they are delivered for tracking purposes to the analysts assigned to this stu dy, Jason Minser. Tracking cases in this manner allows for easy identification of households returning completed surveys, partial surveys and non

- responding households. This process will help us identify mailing method improvements from the pilot to the ma in study, investigate response rates in the pilot study to inform participation in the main study, and generate a list of non

- responding households at the end of the main study to be used in the non

response survey.
Coding/Editing

SODP staff have extensive experience editing/coding paper instruments for the private sector as well as public sector studies that contain confidential materials that may require special handling. Because the majority of this study will be conducted via mail surveys, SOD P will pay additional attention to coding and editing of the surveys in preparation for data entry.

Trained SODP staff will review and prepare for data entry each received and batched survey. The review process will include but will not be limited to con firmation of completion, questionnaire logic (e.g., did they answer the right questions), clarity of response, numeric code imputation (e.g., a number by a check box) and open end coding.

In addition to coding of mail responses, we will also code “open

- e nd” and “other, specify” responses for Computer Assisted Telephone Interviewing internally using Ascribe software. SODP will do this for the non

- response phase of the study. SODP will develop open ended schemas in conjunction with the analysts once 100 com pleted surveys are returned and data are entered. In this way we do not predefine codes and limit response options for survey participants. Once enough surveys are returned and data are entered, the analysts assigned to this study will assign the coding sc hema to be used for the rest of the surveys.

Supervisory staff will perform frequent QC checks to ensure that established editing and coding conventions are followed and strict security and document control measures are maintained throughout all the proc esses.

As part of the training for this project, supervisory staff will explain editing rules for each question

, and will read aloud and explain open

- ended post

- code lists. Editing rules for this survey will consist of how to address multiple responses i n single response questions and extraneous markings. In this training, each editing/coding staff will independently complete the editing/coding of 5

- 10 documents. A senior coder/data coordinator will then perform a Quality Check to ensure that the editing instructions were correctly followed and that postcodes were accurately applied. The supervisory staff will thoroughly review e ach question to ensure that editing rules were correctly applied and then give immediate feedback to coding staff. The senior cod er/data coordinator will use a QC report to document error rate.

In order to perform proper Quality Assurance of open

- end postcodes, the person performing the Quality Check will select codes without looking at those previously applied. The coder assignedAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 5



### to do the Quality Check will then check for agreement and review any discrepancies with a particular coding staff member to gain a better understanding of the code list. For example, a coder will assign codes to a set of open ended responses, followed by a nother coder who will code the same set of responses separately. The person performing the Quality Check will then compare the two sets of codes in order to assess if there are differences in the codes assigned to the same set of responses. If coding staff have difficulties understanding the post

- code list, a senior coder will further check the list to assess whether there is a problem with the post

- code list itself and not with the staff.

In addition to the initial check, for the first day of editing/cod ing, the supervisory staff will QC 100 percent of each coder’s work as soon as possible. If this first

- day review identifies problems, then the supervisory staff will continue to review 100 percent of the coder’s work and give additional feedback to the co der. If the level of QC is continuously met, the checks can be progressively reduced by 20 percent depending on the level of staff competency or project requirements, such that supervisory staff will reduce 100 percent checks to 80 percent, and so on. The staff supervisor will make this determination based on consistent and continuous high quality performance of each staff person. If errors remain frequent for a particular staff person, then the staff supervisor will inform SODP manager and SODP manager wil l remove the specific staff from the project. If removal is warranted, supervisory staff will quality

- check all of the work completed by the staff before data entry.

All of these QC procedures for editing and coding will allow s upervisory s taff to inspect the work of the e diting/coding staff in order to identify repeated errors, identify employees whose work quality is insufficient, identify inconsistencies in project design/rules, aid in the ease of data entry and data cleaning, minimize error

, and greatl y improve the overall quality of the data collected.

Data Entry

Once SODP completes the editing/coding of the paper surveys and the required level of QC is achieved

, SODP supervisor will transfer the batches to the data entry stage. SODP will send each group of forms transferred with a transmittal coversheet identifying the batch number and total number of surveys.

For the Fish and Aquatic Habitat Survey, we will use the Double Entry data entry method for closed

- ended responses. The Double Entry method consists of data being keyed twice and compared. Data entry supervisor will remove discrepancies upon completion of the second entry. Upon review of the hard copy of each survey an operator determines which entry was correct and makes necessary changes. The Double Entry method is optimal for short field periods, high

- levels of data consistency and protocol adherence and when data will be used for subsequent components. The Double Entry method allows for immediate inspection of operator entry errors (e.g., repeated offenses, inaccurate coding); sufficiency tracking of operator rates/quality; and checking for project design/rule inconsistencies. This approach is most effective in minimizing error. For any open

ended responses, we will use single data entry, with the text being entered verbatim.

As with the editing /coding QC measures, as soon as a data entry staff person completes entry of a batch of responses, a supervisor will verify/double enter the batch and provide feedback.Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 6



### For the non

- response portion of the study, the questionnaire will be programmed for data capture utilizing Teleform scanning software with a separate set up required for each of the study regions. Upon return through USPS, questionnaires will be sorted (separated by version/region), batched, and electronically receipted by ID #, batch #, and date received into an excel spreadsheet. Questionnaires will then be manually reviewed/edited in order to correct light fill

ins as well as other infrequent miscellaneous resp ondent error . Data entry for the non

- response study will be accomplished utilizing scanning hardware and software. During the scan verify process operators will review data, enter handwritten data, and correct any errors detected by the software. To ensure data quality, several hard copy cases will be reviewed against data output. Data frequencies will then be reviewed to identify any possible errors and the data will be compiled/delivered in one data file per survey version.”

The general supervisor for QC of mailing, printing, receipt, coding, and data entry is Kevin McMahon.

Final Data Preparation

After Abt SRBI staff enter all data into the database, named WQ

- 4

- 05_T5

_ FishHabitat _Month.Date. Year , and generate data files, senior project analysts assigned to this study

– Jason Minser

– will begin the final check of all entered data using SPSS. This data cleaning step involves a comprehensive review of all data fields for consistency with survey prot ocols. The checks performed during this process will include but are not limited to: ensuring regional version variation is appropriate, identification of unallowable values, checking for non

duplicative unique ids version consistencies, incorrect implemen tation of skip patterns, and data entry errors.

Confidentiality

Abt SRBI’s SODP Center operates within a secured office environment. Entry into the space is possible by two methods: keyed entry and an electronic door release system. All daily traffic

mus t go through the main entrance. All staff members take the utmost precautions to ensure the privacy of individuals responding to surveys and other modes of data collection. Data collected through surveys and other modes of data collection are stored on dat a servers which are under lock and key. Moreover, locked storage areas within the data preparation center safeguard "hardcopy" written responses and information provided by all respondents. All project and client

- provided information is considered propriet ary and is not to be shared without the expressed written consent of the client.

3. TELEPHONE INTERVIEWS

To conduct the telephone interview portion of the non

- response study

, Abt SRBI will administer a short questionnaire by phone. T he Abt WAM for the Stated Preference Survey task , Dr. Besedin , will provide t he phone questionnaire. The following sections describe quality assurance procedures for the phone interviews.

Interviewing Staff

The quality of the interviewing staff used on a survey is one of the most important factors affecting the validity, reliability and timeliness of the data collected. Abt SRBI’s telephoneAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 7



### interviewing staff consists of more than 500 telephone interviewers, who are employed on a full or part

- time basis by our firm.

All interviewers working for Abt SRBI are thoroughly trained and closely supervised. Special training sessions are undertaken for each new project to help ensure Q uality C ontrol (QC) over the survey data collection

. Abt SRBI thoroughly screens new inter viewer s, checks their references, and tests their interview er abilities before they are hired as Abt SRBI telephone interviewers. All interviewers receive extensive instruction in the methods and procedures to be followed at Abt SRBI before they perform intervi ews. Supervisory staff monitor new Abt SRBI interviewers extremely closely during the first two weeks of their employment.

Supervisory staff monitor the performance of all our interviewers, whether they are new or have been with Abt SRBI for a longer per iod, on a regular basis twice per shift.

Employment Characteristics

Number of interviewers: 576

Training: Basic indoctrination training and study specific training

Trial period: Three shifts

Required shifts for interviewers: Minimum of three per week

Monitoring: Minimum of twice per shift

Full review: Every two weeks

Average experience: 1.8 years

Turnover rate: Less than 10% per month

Experienced interviewers: Four years

Spanish

- speaking: Yes, more than 40 on staff

Training of Telephone Interv iewers

All interviewers working for Abt SRBI receive thorough train ing both in general interviewing procedures and the protocol s for a specific study . Abt SRBI interviewers follow a general manual on interviewing procedures developed by our operations sta ff. Supervisory staff may emphasize and recapitulate interviewing procedures for specific projects. Interviewers are trained in:

 An understanding of sampling procedures, generally, and the importance of adhering rigorously to these procedures in the fiel d;  An understanding of respondent selection procedures within a unit and the importance of following them as rigorously as the between

- unit sampling procedures;  The role of the interviewer in the survey process;Abt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 8



###  Recommended methods for contacting potential respondents and procedures for setting appointments;  Effective methods for gaining initial agreement to be interviewed;  Methods for overcoming initial reluctance to schedule or agree to be interviewed;  Interviewer behavior in the interview setting

– how to be courteous, neutral and non

intrusive;  How to avoid biasing responses by verbal and non

- verbal cues;  How to ask and record closed

- ended questions;  How to probe and record open

- ended questions;  How to control irrelevancies and digressions without offen ding the respondent; and,  How to reassure respondents about the confidentiality of the information collected and the anonymity of survey respondents.

Because EPA’s non

- response survey requires additional emphasis and more specialized training in the areas of overcoming initial reluctance to participation, interviewer behavior and recording of closed

- ended questions, Abt SRBI will train the interviewers in these areas as well.

In addition to focused training procedures, the field director in coordination with the project director will develop special training materials. Field kits will include: item

- by

- item interviewing specifications; decision algorithms for contact and location procedures; procedures to maximize the probability of respondent participatio n; proper CATI recording procedures; and additional reporting and quality control requirements for the study. Supervisory staff will monitor interviewers assigned to this project to ensure full compliance with general and emphasized training procedures. Ab t SRBI will provide materials for this study to the EPA.

Supervisory Staff

Training does not start and stop with Abt SRBI interviewers. Abt SRBI telephone supervisors are instructed by field directors on specific study details, undergo additional review of interviewing procedures, both general and specific, and are fully involved and present in project training session s

.
Abt SRBI telephone surve ys rely on two types of supervisors: shift supervisors and line supervisors or monitors. A shift supervisor works closely with the project director and the design team in developing project

- specific field procedures. Shift supervisors are on duty each of t he fourteen weekly shifts. Shift supervisors are responsible for meeting all production requirements. They oversee QC, produc tion rates, and supervising the monitors.

Line supervisors, or monitors, are responsible for directly over seeing individual int erviewers. They audio

- monitor the interviews being conducted, and evaluate performance of the interviewers in conducting the interview on question reading verbatim, pronunciation, proper entry of response sets and ability to build rapport with respondents.

All of the shift supervisors currently working for Abt SRBI have more than seven years of experience as telephone interviewers and supervisors

, and are among the best in survey research.

Monitoring of Telephone InterviewersAbt Associates Inc., November 3 , 2011 Contract EP

- C

- 07

- 023, Assignment 4

- 05, QAPP, Appendix 1 9



### In this study, we will con duct a short telephone interview for main study non

- respondents. A line monitor will silently monitor e ach interviewer at least twice during each interviewing shift. The line monitor will evaluate the interviewer on his or her performance and discusses wit h the shift supervisor any problems that an interview er ha s

. Before the end of the interview shift

, the line monitor a nd/or shift supervisor will discuss the evaluation with the inter viewer. If the interviewer cannot meet Abt SRBI standards, he or she wi ll be released from work

.
All interviewers will undergo two types of monitoring. The study monitor will periodically review the performance of each interviewer during each interviewing shift via computer screen (CRT) and audio monitoring: the monitor will see on the CRT what the interviewer has recorded, while simultaneously hearing the interview via the audio monitor. The audio monitoring will allow the supervisor to determine the quality of the interviewer’s perfor mance in terms of: (1) initial cont act and recruitment procedures; (2) reading the questions, fully and completely, as written; (3) reading response categories, fully and completely, (or not reading them) according to study specifications; (4) whether or not open

- ended questions are properl y probed; (5) whether ambiguous or confused responses are clarified; (6) how well questions from the respondent are handled without alienating the respondent or biasing his/her response; (7)avoiding bias by either comments or vocal inflection; (8) ability to persuade wavering, disinterested or hostile respondents to continue the interview; and (9) general professional conduct throughout the interview.

The supervisor will also monitor the interviewer’s recording of survey responses on the CRT. The superviso r’s CRT will show the interviewer’s CRT. Consequently, the supervisor will be able to see whether the interviewer enters the correct code, number or verbatim in response to the answer. A supervisor will monitor each interviewer at least twice per shift. Th e proportion of interviews monitored for a given interviewer may be increased for a number of reasons. If a supervisor observes problems in an interview, he/she will intensify the interviewer’s monitoring to better understand the problem. If an interviewer attains a higher than average production rate or response rate, the supervisor will automatically intensify monitoring to be sure every interview is done according to specifications. The supervisor will carefully record the results of all study monitoring in the monitoring log for each shift.