## Programmatic Quality Assurance Project
Plan (PQAPP) for Economic, Environmental, and Regulatory Analytical and Evaluation Support for Clean Water Regulations (Contract EP

- C

- 13

- 039)

Submitted to :

U.S. EPA/OW/OST/EAD (4303T) EPA West, Room 6233D 1200 Pennsylvania Ave., N.W. Washington, DC 20460
Attn: Marion Kelly Kelly.M arion@ epa.gov

Submitted by:

Abt Associates
55 Wheeler Street Cambridge, MA 02138

Contract No. EP

- C

- 13

- 039

July 31, 2014Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 i



## Revision History

Revision Number Date

Submitted Date Approved Record of Activity and Scope of Revisions

0 7/31/14 8/7/14 Original P QAPPAbt Associates PQAPP EP

- C

- 13

- 039 July 2014 ii



## Title and Approval Sheet

Project Title: Programm atic Quality Assurance Project Plan (PQAPP) for Economic,

Environmental, and Regulatory Analytical and Evaluation Support for Clean Water Regulations ( Contract EP

- C

- 13

- 039 )
Organization Name: Abt Associates Inc.

Date: July 2014

Revision Number: 0
A signature b elow indicates each individual’s approval of this Plan

, and commitment to follow the procedures it describes.



### Abt Associates Approval Signatures

7/31/14

Michael Fisher

, Project Manager , Contract No. EP

- C

- 13

- 039 Date

7/30/14 Susan Day, Quality Assurance Manager, Contract No. EP

- C

- 13

- 039 Date

7/30/14
Eloise Castillo, Deputy Work Assignment Manager, No. 0

- 01 Date

7/30/14

Anna Belova, Project Qua lity Assurance Advisor Date

7/30/14
Isabelle Morin, Project Quality Assurance Advisor Date

7/30/14

Viktoria Zoltay, Project Quality Assurance Advisor Date

### Environmental Protection Agency Approval Signatures

Marion K elly, Work Assignme Manager

, Engineering and Analysis Division Quality Assur ance O ffice r



### t()

## ~**~~ ~**  Ahmar Siddiqui, Project o lffC er , Contract No

. EP-C-13-039

Juli e J;lewi tt

, E conomic arid · E nvironmental Assessment B r anch Chief**------------------~-------** Ab t Associa t es P Q APP E P

- C- 1 3

- 0 39 

## v'1: ( o · r/ ?»'If

Date

July 2014 ii iAbt Associates PQAPP EP

- C

- 13

- 039 July 2014 iv



## Distribution List

EPA and Abt Associates will distribute the approved P QA PP and any subsequent revisions to the following individuals:
U.S. Environmental Protection Agency [ Engineering Analysis Division , Office of Science and Technology, 1200 Pennsylvania Avenue, NW (4303T), Washington, DC 20460] :

 Marion Kelly, EAD Quality Assurance Coordinator, (202) 566

- 1045

, Kelly.Marion@epa.gov

 Ahmar Siddiqui , Project Officer, Contract No. EP

- C

- 13

- 039 ; (202) 566

- 1044

, Siddiqui.Ahmar @epa .gov
 Julie Hewitt, Economic and Environmental Assessment Branch Chief , (202) 566

- 1031, Hewitt.Julie@epa.gov

 Work Assignment Manager, specified in Work Assignment

Ab t Associates Inc

. ( Offices at: 55 Wheeler Stre et, Cambridge, MA 02138

- 1168 ; and 4550 Montgomery Avenue, Suite 800 North, Bethesda, MD 20814 ) :
 Michael Fisher, EPA Contr act EP

- C

- 13

- 039 Project Manager , (617) 349

- 2526, Michael_Fisher@abtassoc.com

 Susan Day, EPA Contract EP

- C

- 13

- 039 Quality Advisor, (617) 349

- 2768, Susan_Day@abtassoc.com

 Elena Besedin, Senior Benefits Economist Key Personnel , (617) 349

- 2770, Besedin_Elena@abtassoc.com

 Isabelle Morin, Senior Environmental Assessor Key Personnel, (617) 349

- 2667, Isabelle_Morin@abtassoc.com

 Eloise Castillo, Senior Cost Economist Key Personnel , (301) 347

- 5701, Eloise_Castillo@abtassoc.com

 Anna Belova, P roject Q uality A ssurance Advisor, (301) 347

- 5304, Anna_Belova@abtassoc.com

 Victoria Zoltay, Pr oject Quality Assurance Advisor, (617) 520

- 2721, Viktoria_Zoltay@abtassoc.com

 Dan Basoli, Work Assignment Manager , (617) 349

- 2344 , Dan_Basoli@abtassoc.com

 Debra Fleischer, Work Assignment Manager, (301) 347

- 5396, Debra_Fleischer@abtassoc.com

 Emily Giovann i, Work Assignment Manager, (301) 347

- 5398, Emily_Giovanni@abtassoc.com

 Tatiana Melnikova, Work Assignment Manager, (617) 520

- 2824

, Tatiana_Melnikova@abtassoc.com

 James Palardy, Work Assignment Manager, ( 301) 347

- 5235, James_Palardy@abtassoc.com

 Ryan Stapl er, Work Assignment Manager, (617) 520

- 3524

, Ryan_Stapler@abtassoc.comAbt Associates PQAPP EP

- C

- 13

- 039 July 2014 v



## Programmatic Quality Assurance Project Plan Content

Exhib it 1 provides a crosswalk of the Programmatic Quality Assurance Project Plan to elements

identified in EPA Requirements for Quality Management Plans (QA/R

- 2) , March 2001 (Reissued May 2006)

.

Exhib it 1 : Crosswalk of QAPP elements to PQAPP section.

E lement Page/ Section

Group A. Project management

A1. Title and Approval Sheet P age ii

A2. Table of Contents Page vii

A3. Distribution List P age iv
A4. Project/Task Organization Section 2.1

A5. Problem Definition and Background Section 2.2
A6. Project/Task Description Section 2.3

A7. Overal l Quality Objectives and Criteria Section 2.4
A8. Special Training/Certifications Section 2.5

A9. Project

- Level Documentation and Records Section 2.6

Group B. Data Generation and Acquisition

Existing Data Gathering Project

B1. Data Needs Section 3.1.1

B2. Potential Data Sources Section 3.1.2
B3. Criteria for Selecting Data Sources Section 3.1.3
B4. Data Value Selection Approach Section 3.1.4
B5. Resolving Data Gaps Section 3.1.5
B6. Data Gathering Documentation and Records Section 3.1.6
Primary Data Gathering Project

B1. Sampling Process (Experimental) Design Secti on 4.1.1
B2. Sampling Methods Section 4.1.2 (Data Gathering Methods and Steps)

B3. Sample Handling and Custody Section 4.1.3 (Completed Questionnaire Handling and Custody)

B4. Analytical Methods Not applicable 1

B5. Quality Control Not applicable 1

B6. Instrument/Equipment Testing, Inspection, and
Maintenance Not applicable 1

B7. Instrument/Equipment Calibration and Frequency Not applicable 1

B8. Inspection/Acceptance of Supplies and
Consumables Not applicable 1

B9. Non

- direct Measurements Section 3.1.2

B10. Data Management Section 4.1.4

G roup C. Data Management

Existing Data Gathering ProjectAbt Associates PQAPP EP

- C

- 13

- 039 July 2014 vi

C1. Standardization of Data Elements Section 3.2.1
C2. Data Entry Section 3.2.2
C3. Merging or Uploading Electronic Data from Existing Sources Section 3.2.3

C4. Data Review Section 3.2.4
C5. Data Storage and Manipulation Section 3.2.5

Primary Data Gathering Project

C1. Assessments and Response Actions Section 4.1.5
C2. Reports to Management Section 4.1.6

Group D. Data Quality Assessment and Us e
Existing Data Gathering Project

D1. Data Quality Verification and Data Quality Reporting Section 3.3.1

D2. Use/Analysis of the Existing Data Section 3.3.2
D3. Methodology Documentation and Conceptual
Review Section 3.3.3

D4. Technical Review of the Data Analysis Section 3.3.4

D5. Final Verification of Data Analysis and Reconciliation with User Require ments Section 3.3.5

Primary Data Gathering Project

D1. Data Review, Verification, and Validation Section 4.2.1

D2. Verification and Validation Methods Section 4.2.1

D3. Reconciliation with User Requirements Section 4.2.2

1. Project does not involve laboratory or field measurements.Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 vii



## Contents

1. Introduction

................................ ................................ ................................ ............................... 3

2. Project Management

................................ ................................ ................................ ................. 5 2.1 Project Organization

................................ ................................ ................................ .......... 5
2.2 Problem Definition/Background

................................ ................................ ........................ 9
2.3 Project/Task Description

................................ ................................ ................................ . 14
2.4 Overall Data Quality Objectives and Criteria

................................ ................................ .. 16
2.4.1 Environmental Assessments

................................ ................................ ............... 16
2.4.2 Economic/Financial Analyses

................................ ................................ ............. 17
2.4.3 Benefits Valuation

................................ ................................ .............................. 19
2.5 Training and Certification

................................ ................................ ................................ 22
2.6 Project

- Level Documents and Records

................................ ................................ ............ 23
2.6.1 CBI

................................ ................................ ................................ ..................... 24
2.6.2 Deliverables

................................ ................................ ................................ ........ 24
2.6.3 Qual ity Audits

................................ ................................ ................................ .... 25

3. Existing Data

................................ ................................ ................................ ............................ 27 3.1 Data Gathering

................................ ................................ ................................ ................ 27
3.1.1 Data Needs

................................ ................................ ................................ ......... 27
3.1.2 Potential Data Sources

................................ ................................ ........................ 40
3.1.3 Criteria for Selecting Data Sources

................................ ................................ ..... 45
3.1.4 Data Value Selection Approach

................................ ................................ .......... 48
3.1.5 Resolving Data Gaps

................................ ................................ .......................... 49
3.1.6 Data Gathering Documentation and Records

................................ ...................... 49
3.2 Data Management

................................ ................................ ................................ ............ 51
3.2.1 Standardization of Data Elements

................................ ................................ ....... 51
3.2.2 Data Entry

................................ ................................ ................................ ........... 52
3.2.3 Merging or Uploading Electronic Data from Existing Sources

........................... 52
3.2.4 Data Review

................................ ................................ ................................ ....... 53
3.2.5 Data Storage and Manipulation

................................ ................................ .......... 53
3.3 Data Quality Assessment and Use

................................ ................................ ................... 54
3.3.1 Data Quality Verification and Data Quality Reporting

................................ ....... 54
3.3.2 Use/Analysis of Existing Data/Data Operations

................................ ................. 54
3.3.3 Methodology Documentation and Conceptual Review

................................ ....... 56
3.3.4 Technical Review of the Data Analysis

................................ .............................. 57
3.3.5 Final Verification of Data Analysis and Reconciliation with User Requirements 57

4. Primary Data

................................ ................................ ................................ ............................ 59 4.1 Data Gathering

................................ ................................ ................................ ................ 60
4.1.1 Samp ling Process (Experimental) Design

................................ ........................... 60
4.1.2 Data Gathering Methods and Steps

................................ ................................ ..... 61
4.1.3 Completed Questionnaire Handling and Custody

................................ ............... 62
4.1.4 Data Management

................................ ................................ ............................... 63
4.1.5 Assessment and Response Actions

................................ ................................ ..... 64
4.1.6 Reports to Management

................................ ................................ ...................... 64
4.2 Data Quality Assessment and Use

................................ ................................ ................... 64
4.2.1 Data Review, Verification, and Validation

................................ ......................... 64
4.2.2 Reconciliation with User Requirements

................................ .............................. 65
4.3 Telephone Interviews

................................ ................................ ................................ ...... 66Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 viii

5. References

................................ ................................ ................................ ............................... 6 7Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 i



### Abbreviation s

BFT Benefit Function Transfer

BMP Best Management Practice

BUVD Benefit Use Valuation Database

CATI Computer Assisted Telephone Interviewing

CBI Confidential Business Information

CWA Clean Water Act

EAD Engineering and Analysis Division

EIA Energy Information Administration

ELG s Effluent Limitation s Guidelines

EPA Environmental Protection Agency

ESD Environmental site design

EVRI Environmental Valuation Resource Inventory

FAQ Frequently Asked Questions

HUC Hydrologic Unit Code

ICIS Integrated Compliance Information System

ICR Information collection request

IRIS Integrated Risk Information System

IT Information Technology

LID Low impact development

NAICS North American Industry Classification System

NPDES National Pollutant Discharge Elimination System

OST Office of Science and Techno logy

OW Office of Water

PDF Portable document format

PQA Project Quality A ssurance
PQAPP Programmatic Quality Assurance Project Plan

PWS Performance Work Statement

QA Quality assurance

QA/QC Quality assurance and quality control

QAC Quality A ssuran ce C oordinator

QAPP Quality Assurance Project Plan

QMP Quality Management Plan

RFA Regulatory Flexibility Act

SEC Securities and Exchange Commission

SQAPP Supplemental Quality Assurance Project Plan

SWAT Soil and Water Assessment ToolAbt Associates PQAPP EP

- C

- 13

- 039 July 2014 i i

UMRA Unfunded Mandates Reform Act

USGS United States Geological Survey

WAM Work Assignment M anager
WQI Water quality index

WQL Water quality ladder

WSUT Weak structural utility theoretic

WTP Willingness to pay

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 3 

### 1. Introduction

This Programmatic Quality Assurance Project Plan (PQAPP) covers quality assurance and quality control (QA/QC) applicable to the collection, generation, evaluation, analysis, and use of environmental data by
Abt Associates Inc. to provide Economic, Environm ental, and Regulatory Analytical and Evaluation Support for Clean Water Regulations (Contract EP

- C

- 13

- 039).

The Office of Water (OW) of the United State s Environmental Protection Agency ( EPA , or the Agency)
is responsible for implementing the Clean Water A ct (CWA). The Engineering and Analysis Division (EAD) within OW, and the Office of Science and Technology (OST)

, develops national industrial discharge regulations known as effluent guidelines. The primary purpose of th e Performance Work Statement (PWS) fo r Contract EP

- C

- 13

- 039 is to support economic and e nvironmental analyses related to the development of effluent guidelines regulations. This support encompasses:

 Environmental asses smen t

 Economic and financial analysis

 Environmental benefits valuation.

O th er divisions within the OW that use this contract to support their programmatic activities may need to develop a project

- specific Quality Assurance Project Plan (QAPP)

. EPA requires, through its quality assurance (QA) policy, that a QAPP be approved prio r to commencing
work that involve s the generation, collection, analysis, and use of environmental data

. Under Contract EP

C

- 13

- 039, Abt Associates Inc. (Abt Associates) will perform work that involves the se types of environmental data operations ( see Table 1

- 1 )

. For example, support to EPA in analyzing the economic impacts of regulatory options may entail a wide range of data

- related activities. These activities may include generation of primary da ta by surveying affected entities, collection of existing data from public sources, evaluation and assessment of data to ensure completeness and representativeness, analysis of the impacts of increased costs on various entities, and use of the data to supp ort EPA in making decision s related to implementing the CWA, Executive Orders

, or other requirements. T o facilitate compliance with the policy, and to assist EAD in ensuring the quality of the work products , Abt Associates has prepared
this PQAPP for work involving both primary and existing environmental data. Primary data are data generated by EAD or under its direction to support the decision at hand ; existing data are any data not generated directly by EAD to support the decision at hand

.

Table 1

- 1 : Examples of work that involve the collection, generation, evaluation, analysis, or use of environmental data.

Item Examples

Data generation Field studies, laboratory studies, and modeling output generation

Data collection Field surveys, questionnaires, literature searches, and third party data collection

Data evaluation Data inspection, review, assessment, and validation evaluation

Data an alysis Statistical, engineering, and economic analysis, and testing, evaluation, and validation of

analysis methods and models; database creation, data extraction, and data manipulation

Data use Support for EPA decisions, regulations, policy, publications, or tools ( including effluent

guidelines, 304(m) program, standards, environmental assessments, and models, tools, or reports disseminated by EPA to assist other organizations in impl ementing environmental programs)

EAD develops effluent guidelines in a progressively elaborative manner as it gathers and evaluates data, and evaluates new needs and gaps . Th us, the activities we describe in this document

, based on past experience and current work assignments

, may not be comprehensiv e of all work undertaken under this

contract

. As needed and directed by EPA, Abt Associates will prepar e supplemental QAPPs (S QAPPs) or

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 4 traditional QAPPs for individual work assignments . SQAPPs supplement the approved PQAPP by providing the additional deta il needed to describe the specific goals and objectives of the project or tasks supported under the work assignment, as well a s the specific QA strategies and technical activities that the project team will use to support the objectives of the work assignm ent project or task. T raditional QAPPs for individual work assignments are stand

- alone documents that focus on the goals, objectives, QA strategies, and technical activities needed to support a single work assignment or project. Abt
Associates will then in corporate changes into th is PQAPP as appropriate.
Abt Associates developed this PQAPP in accordance with EPA ’s Guidance for Quality Assurance Project
Plans (EPA/240/R

- 02/009) , and the suite of alternative elements that EAD designed to improve the quality of QAPPs addressing existing data ( EPA EAD QAPP Guidance )

. All project deliverables that Abt Associates prepares under this contract will meet EPA ’s standards of transparency, reproducibility, objectivity, integrity, and utility as spec ified in EPA ’s Guidelines for Insuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency (EPA/260R

- 02

- 008, October 2002), referred to as “EPA’s Information Quality Guidelines

. ”
The remainder of this document is organized as follows:

 Section 2 describes Project Management QAPP elements (Elements A in EPA guidance)

.  Section 3 describes QAPP elements pertinent to existing data collection, management, assessment, and use

.  Section 4 describes QAPP elements pertinent to primary data collection, management, assessment, and use

.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 5 

### 2. Project Management

This section addresses project management elements , including project organization , roles and responsibilities, and goals and objectives

. In addition, this section presents Abt Associates’ approach for ensuring that all members of the project te am understand the goals and procedures applicable to this

program. Abt Associates meets quality objectives through setting clear proj ect organization and objectives; incorporating quality assurance within all aspects of collecting, managing, and using data ; assigning appropriate staff with the requisite expertise and training ; ongoing verification and review of work products and QA procedures implementation ; and documentation.



### 2.1 Pro ject Organization

EAD will oversee the program, and individual projects

, cover ed by this PQAPP

. Figure 2

- 1 shows the organization and roles within EAD.

Figure 2

- 1 : Project Organizational Chart

.

T he Economic and Environmental Assessment Branch Chief, Ms. Julie Hewitt , will be responsible for oversight of project activities on behalf of EPA . In this capacity, Ms. Hewitt will be responsible for:

 Participating in project planning activities and providing direction regarding project scope,

resource s, design and quality requirements

 Approving the final project design and quality objectives

 Assigning project staff within the EAD Economic and Environmental Assessment Branch, and securing all staff and financial resources necessary to design and implement the project

 Assisting and providing technical input to EPA Work Assignment Manager (WAM) as needed

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 6  Approving this P QAPP , any EAD SQAPPs

, and other EAD quality planning documents including project

- specific QAPPs

 Overs eeing project budgets and sc hedules

. The EPA WAM for a specific assignment will be responsible for day

- to

- day management and direction of all study activities. In this capacity, the EPA WAM will be responsible for:

 Managing the work assignment under which Contractor support will be p rovided, directing all Contractor activities associated with implementation of the project, and ensuring that the Contractor is provided with the resources necessary to implement the project

 Producing (or directing production of) a written project plan tha t details project objectives, tasks, schedules, roles, and responsibilities

 Managing project budgets and schedules, and notifying the Branch Chief of problems or delays that may impact project objectives, budgets, or schedules

 Coordinating with the EAD Qua lity Assurance Coordinator (QAC) to define and document quality objectives and QA/QC strategies for the project and to produce this PQAPP and any SQAPPs, as needed

 Adhering to all applicable EPA requirements concerning the quality of information produced

u nder this project (e.g., OW’s Quality Management Plans, EPA’s Information Quality Guidelines)

 Ensuring that all staff involved with the project (inside and outside of EAD ) adhere to the requirements of this PQAPP.

The EAD QAC, Marion Kelly , will be respons ible for:

 Providing guidance to the EPA W AM concerning the Agency’s quality requirements an d recommending QA/QC strategies

 Assisting the EPA W AM as needed with implementation of QA/Q C strategies

 Re viewing and approving this PQAPP , any SQAPPs, and other qua lity planning documents including project

- specific QAPPs

. The EA D QAC reports to the EAD Director

, and is independent from the technical mana gers who oversee the pro gram and proj ect s

. Figure 2

- 1 also shows Abt Associates’ organization as pertains to QA/QC activity for this contract

. As directed by the EPA WAM

, Abt Associates and its subcontractors and consultants will be responsible for drafting, revising, and/or finalizing study materials. Examples of these materials include this P QAPP, any SQAPP s needed, surveys or questionnaires

, analytic files, memoranda, and a final study report that includes a distinct section addressing QA/QC ( see Section 3.3.5 )

. Abt Associates will provide scientific and statistical support to EPA in resolving problems and evaluating data. As shown in the figure, we have organized our support to ensure the independ ence of the Abt Associates Quality Assurance Manager from the Project Manager.

Table 2

- 1 provides a summary of the roles and QA responsibilities of Ab t Associates staff, with detailed descriptions following the table.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 7 Table 2

- 1 : QA Roles and Responsibilities.

Position Quality

- R elated Responsibility

Project Manager (Michael Fisher)  Provide manag erial oversight of the contract

 Work with the Abt Associates WAM to assign a PQA Advisor

 Review and approve PQAPP

, SQAPPs , and other QA planning documents such as a project

- specific QAPP

Quality Assurance Manager (Susan Day)  I mplement, monitor, and maintain the procedures described in the Quality Management Plan, PQAPP, SQAPPs , and other QA planning documents such as a project

- specific QAPP

 Ensure qualified Project Quality Assurance Advisors assigned to every WA

Project Quality Assurance
Advisor (TBD on project

- specific basis)  Ens ure and enhance project quality

 Provide input for SQAPP or other QA planning document such as a project

- specific QAPP (if needed), and approve

 T echnical review

Work Assignment Manager (TBD on project

- specific basis)  Develop and approve S QAPP or other QA planning document such as a project

- specific QAPP (if needed)

 Work with the Project Manager to identify a P roject Q uality Assurance
Advisor

 O rganize the work and apply resourc es to accomplish the objectives

 Monitor work, schedules, and budgets

 Ensure sufficient time and resources are allocated to QA/QC procedures

 Ensure project team members, including subcontractors and consultants, are aware of and implement PQAPP and SQAPP procedures

 Report on QA/QC activities

Modeler, Spreadsheet or Database Developer (TBD)  Review and adhere to PQAPP , and SQAPP or other QA planning document such as a project

- specific QAPP (if needed)

 Develop model, spreadsheet, or database development following best practices

 Participate in routine and ad

- hoc conference calls to discuss project status, issues, and questions

 Notify Abt Associates Work Assignment Manager of issues that will affect project budgets, schedules, or goals

Additional Project Team
Member s (TBD on project

- specific basis)  Review and adhe re to PQAPP

, and SQAPP or other QA planning document such as a project

- specific QAPP (if needed)

 Participate in routine and ad

- hoc conference calls to discuss project status, issues, and questions

 Notify Abt Associates Work Assignment Manager of issues tha t will affect project budgets, schedules, or goals
PQA = Project Quality Assurance PQAPP = Programmatic Quality Assurance Project Plan QA = quality assurance QA/QC = quality assurance/quality control SQAPP = Supplemental Quality Assurance Project Plan

TBD = to be determined WAM = work assignment manager

Mr. Michael Fisher , the Abt Associates Pro ject M anager , will manage the activities by Abt Associates
and its subcontractors and consultants. Mr. Fisher reviews and approves this PQAPP, and any SQAPPs prepared for individual WAs. He also assigns the Abt Associates WAM, and works with the W AM to

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 8 designate the assignment

- level Project Quality Assurance (PQA) Advisor . If any QA concerns arise in the
course of the work that cannot be resolved within the pro ject team, Mr. Fisher will refer these concerns to
the Contract

- Level Quality Assurance Manager

, Ms. Susan Day

. Ms. Susan Day maintains overall responsibility for implementing, monitoring, and maintaining the procedures described in the Abt Quality Manage ment Plan (Q MP ) as they apply to this contract (including ensuring that the Project Manager and WAMs have assigned a qualified PQA Advisor for each work assignment ), reviews and approves the PQAPP and any SQAPPs, and verifies that WAM s and PQA
Advisor s adh ere to procedures outlined therein . Ms. Day is organizationally separate from Mr. Fisher, the WAMs, and the day

- to

- day work on the project s , reporting directly to Andrew Stoeckle

, Vice
President/ Principal Associate

. The Abt Associates WAM works with the Project Manager to identify and assign a technically qualified individual as the PQA Advisor . The Abt Associates WAM also typi cally develops the S QAPP

, if needed

. Depending on the nature of the work assignment , the W AM may work with a subcontractor or cons ultant to develop the S QAPP , or parts thereof

. The WAM ensures that all staff working on the individual project s are aware of the P QAPP , and have received a copy of the document.

T he Abt Associates WAM writes, or directs the writing of a P QA Plan. The P QA Plan describes the key steps in the project, the level of PQA Advisor involvement, and the project technical review requirements. Susan Day, the Quality Assurance Manager for this contract, and Abt Associates’ division management must approve the plan; t he WAM is responsible for allocating appropriate time for review of work products into the project or task schedule, and for dedicating adequate resources from the work assignment budget for technical review and QA oversight.

At the start of each project, a nd at various times during project execution, the Abt Associates W AM
reviews the QA procedures with the task leads to confirm that the leads are implementing the procedures

, and discuss any needed improvements. To the extent that such reviews identify need ed changes in procedures for conducting analyses or conducting QA/QC, we will discuss these changes with the EPA

WAM . We note these discussions and changes in monthly project status reports to the EPA WAM , and may incorporate the revised procedures in futu re versions of this PQAPP.

The Abt Associates PQA Advisor ensures and enhances project quality by providing project staff access to the experience and judgment of a senior staff member who is not immediately involved in the project work, and by providing ongoing independent technical review of project work.
The PQA Advisor participates in quality enhancement discussions with the Abt Associates WAM or other project staff at several key points during the project (e.g., see Section 2.6.3 on qual ity reviews) . These discussions provide the project staff with the benefits of an objective assessment of plans, progress to date, and potential pitfalls. The extent and nature of such discussions depend on the complexity of the project, the potential for quality concerns, and the cost associated with waiting to address problems until

the technical review.
The PQA Advisor also provides technical review of project work. The technical review requirement is essentially a peer review requirement. This requirem ent is consistent with the nature of our work, which is
innovative and project

- specific rather than routine

, and often involves considerable expertise and judgme nt in applying that expertise. In general, the Abt Associates WAM may not submit any substantiv e project deliverable or product to a client in either draft or final form without the technical reviewer’s certification that it is technically acceptable in terms of content, methods, and exposition. For example, in reviewing a report, the reviewer usual ly reviews the draft, marks it up, and discusses problems and comments with the authors. Reviewers are concerned not only with methods and interpretation, but also with the clarity of the presentation and the application of the findings to the client’s pro blem. The reviewer rereads all or part of the revised draft as necessary

, and must approve the draft before we send it
to the client . (If there are any unresolved quality issues following this review, the PQA Advisor consults

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 9 with t he Quality Assurance Man ager for the contract, Susan Day, who may involve Division management in resolving the issues.) The project schedule must provide time for both review and response to reviewer

comments.
Technical review for projects may also include after

- action reviews. T hese review s enhance future performance on similar projects through review of the challenges and accomplishments of completed projects. As described above, to the extent that such reviews identify needed changes in procedures for conducting analyses or conducting QA/QC, we will discuss these changes with the EPA WAM

, and may incorporate the revised procedures in future versions of this PQAPP.

The technical staff for any partic ular WA may include economists, environmental scientists

, and other staff in a range of disciplines and expertise, depending on the work assignment. Staff involved in collecting, processing, and analyzing data have primary responsibility in implementing QA procedures including ensur ing that computer programming and development of models, spreadsheets

, and databases follow best practices , as described in Section 3.2 . Best practices serve to minimize errors and maximize transparency in data development and analysis, and facilitate quality review of the work pr oduct. Staff in these capacities review and adher e to the P QAPP and SQAPP , participat e in confer e nce calls to discuss the work, and notify the W AM of any issues that may affect the project schedules, budgets, and goals.

Abt Associates subcontractors and consultants review and adhere to this PQAPP, and any SQAPPs and traditional QAPPs . These project team members participate in conference calls to discuss the work, and notify the W AM of any issues that may affect the project schedules, budgets, and goals.



### 2.2 Problem Definition / Background
The goal of this program is to conduct e nvironmental and economic analyses that inform EAD’s development of Effluent Limitations Guidelines ( ELGs ) . In conducting environmental analyses, Abt Associates brings EAD access to cost

- efficient modeling and government

- produced data relevant to characterizing pollutant l oadings and reductions. With respect to economic analyses, Abt Associates conducts work under this program pursuant to the Agency’s guidelines, Guidelines for Preparing Economic Analyses , (EPA 240

- R

- 10

- 001; December 2010), Final Guidance for EPA Rulewriter s: Regulatory Flexibility Act as amended by the Small Business Regulatory Enforcement Fairness Act (EPA, November 2006), Interim Guidance on Considering Environmental Justice During the Development of an

Action (EPA, July 2010), and other guidance

. Additio nal considerations for determining data and analytic

needs include the type of demonstration that EPA uses to support policy decisions , depending on regulatory requirements . For example, E AD determine s ELGs based on th e best available technology that

is ec onomically achievable. In conducting environmental analyses, Abt Associates brings EAD access to

cost

- efficient modeling and government

- produced data relevant to characterizing the economic, environmental, and financial impacts associated with pollutant lo adings and reductions.
The CWA directs EPA to promulgate ELGs that are economically achievable based on available technology, by categories or subcategories of industrial point sources. EAD bases t hese ELGs on specific technologies (including process changes) that meet the levels of control prescribed in the CWA. The Act prescribes six levels of control based on the type of source (new vs. existing), the type of discharge (direct vs. indirect), and the type of pollutants being discharged (conventional, non

- conventional, or priority pollutants)

. Chapter 1 of EPA’s Technical Support Document for the Preliminary 2010 Effluent Guidelines Program Plan, October 2009, EPA 821

- R

- 09

- 006 (EPA, October 2009) , provides additional background information about CWA re quirements and the effluent guidelines program.
Develop ing ELGs is a complex

, iterative, multiyear effort that includes several distinct project phases and technical disciplines, as shown in Figure 2

- 2 . In developing these ELGs, EAD receives support from

contractors with needed technical expertise and experience

. This P QAPP describes the activities that Abt

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 10 Associates, and its subcontractors and consulta nts on this contract, will perform in working with
environmental data related to Steps 2, 3, 4, 7, 12 , 13c and 13d, and 14 through 17 in Figure 2

- 2

. Table 2

- 2 summarizes the questions to be answered by the different types of environme ntal assessment, economic/financial, and benefits analyses, and the work activities that produce the answers. Each WA under this contract describes the specific objectives needed to support the s e goal s , including generating, collecting, analyzing, and usin g environmental data to model and assess the environmental impacts of pollution controls, estimate the social costs and benefits associated the controls, and compare the impacts, costs, and benefits of the alternative regulatory options that EAD may consid er

. Appendix A provides the QA checklist for the activities conducted for each of these analyses.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 11 Figure 2

- 2 : Effluent Guidelines Development Flowchart.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 12 Table 2

- 2 : Goals of the Analyses. Analyse s Work Activities Questions Answered

Environmental

assessment  R eview and characterize

the environmental effects of pollutant discharges

 Model the changes in
environmental conditions
result ing from regulatory options  What is the impact of pollutant loadings on human health?

 What is the impact of pollutant loadings on ecosystem services provided by water resources?

 Are there ancillary impacts of pollution controls on air and
terrestrial resources?

 What environmental effects will the options have on affected
waters and other media?

Cost

- benefit
and economic impact analysis  Estimate private and

social costs of regulatory options and compare to
social benefits  Would the options have an annual economic impact of $100,000,000 or more?

 Would the options impose incremental costs to state and local governments?

 Would the options cause some facilities to encounter severe financial stress and close ?

 Would the options cause some facilities moderate financial

stress (increased challenge and cost in accessing financial capital) ?

 Which options are e conomic ally achievab le?

 How many small businesses will incur a significant impact?

 Would a substantial number or percent of small entities incur a significant impact ?

 Would consumer prices increase ?

 How will the options affect employment ?

 Do compliance costs pose barriers to entry for new firms?

 What are the benefits from the options, including primary and secondary?

 How do benefits vary across the o ptions?

 What categories of benefits can be quantified, and monetized?

 What are net benefits across options?

 How are net benefits likely to change with the ab ility to quantify
and monetize all benefits , and what is the “break

- even” level of
benefits ?

Cost

effectiveness analysis  Calculate cost

effectiveness of regulatory options  How do reductions in toxic

- weighted pollutant loadings to surface waters vary with the cost of achieving those reductions

across the options?

 How do the unit costs of achieving toxic

- weighted pollutant discharges compare to previously promulgated Effluent
Limitations Guideline s , considering differences in industrial category and available technologies ?

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 13 Table 2

- 2 : Goals of the Analyses. Analyse s Work Activities Questions Answered

Environmental justice and
other
distributional analysis  Analyze the socio

economic characteristics of populations that incur
the environmental damage s that would be reduced by the regulatory options

 Analyze the distribution
of benefits and price increases among popula tions affected by the regulatory options  What populations are affected by pollutant discharges?

 Are minority, low

- income, and indigenous p opulations (Environment Justice populations) subject to disproportionate environmental and public health impacts?

 How would the options contribute to lessening impacts on these populations ?

 Would the options have different distributions of benefits across the Environmental Justice populations?

 If prices for affected goods and services are likely to increase because of the compliance costs, how would these price
increases and associated increases in consumer costs be distributed among various customer groups ?

 Would increases in consumer costs cause undue economic
burden and/or fall disproportionately on Environment Justice populations?

Industry profiles  Compile and describe

data on facilities and
entities within the industry (ies) affected by the regulatory options  What categories and subcategories of industries discharge the pollutants of concern?

 What are baseline economic and financial conditions , pollution control technologies, and pollutant loadings?

I ndustry financial survey development  Develop questionnaire

and Information Collection Request

 Review and compile survey data  What are the baseline financial circumstances of firms employing different pollution control technologies?

T ravel cost models, hedonic implicit price models,
and stated prefer ence survey research  Develop survey

instruments

 Compile and analyze
data to determine statistically significant factors and relationships  What is the value of environmental goods and services?

 What is the incremental value of water quality improvements?

R esearch and
evaluat ion of
new methods
for economic analysis related
to the control of water pollution  Review and compile

l iterature, models, and existing data  Are there i nnovative or streamlined approaches to conduct ing

cost

- benefit analysis, economic impa ct analysis, cost

effectiveness analysis, equity analysis of effluent guidelines and standards, or other regulatory and nonregulatory programs and initiatives ?

Life cycle analysis  Analyze data to identify

sources of pollutants of

concer n to receiving

waters  Are there p ollution prevention opportunities ?

 What is the merit of r egulatory and nonregulatory options involving transfer of pollutants from one media to another ?

Collection/ preparation of reports  Prepare draft and final

Regulatory Impact Analysis, Benefit

- Cost Analysis, and
Environmental
Assessment and other reports  Do the d ata, methods, quality assurance/controls, and results of studies and analyses support the decisionmaking in the regulatory development process?

 Is the decision

- m aking in support of the regulatory development

process transparent in the record?

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 14 Table 2

- 2 : Goals of the Analyses. Analyse s Work Activities Questions Answered

Review and analysis of public

comments  Compile, parse, and

categorize comments

 Draft responses to
substantive comments  Impact of data and analyses provided by public on EPA ’ s draft regulatory selection decisions

Legislative and litigation support  Summarize data,

methods, and key findings  Impact of alternative legislative proposals

 Response to allegations relating to rulemaking or planning
record

Database development
and management  Assemble information in

a systematic relational
framework  How can databases containing d ischarge, environmental, and economic characteristics of the sectors potentially subject to regulation/control be developed and managed to ensure the

accurac y, reliability, and integrity of the data and information?

Outreach  Draft outreach

documents such as web pages, fact sheets, rule summaries, etc.  Is the regulatory process transparent to stakeholders?

Training and logistical support  Organize and facilitate

meetings and training

events  How can stakeholders be trained to implement the rulemaking efficiently?



### 2.3 Project/Task Description
Abt Associates typically conducts analyses under this pro gram starting with an understanding of the entities (e.g., plants, sites) and waterbodies that a given policy would affect, and the associated baseline conditions relative to the incremental effects of the policy that we evaluate. We then superimpose the
effects of the policy on baseline conditions to estimate the changes attributable to the policy.

A ccordingly, in gathering data, we see k to provide a comprehensive understanding of affected entities
and/or waterbodies. When the univer se of affected entities is well

- defin ed and the number relatively small (e.g., a few hundred plants can be readily identified through public sources), we often gather information about each individual entity. For example, we may gather data for electricity generating plant s from public source s such as Department of Energy databases or through surveys of plants to support analysis of regulations affecti ng the electric power industry.

When the universe of affected entities is less well

- defined or the number of dischargers is very large (e.g.,

t housands of construction sites or diffuse nonpoint sources), we may instead gather data for a representative sample of dischargers or locations that cover a range of operating conditions for the industry. For example , analysis of the effects of stormwater management requirements may consider costs and environmental effects for a diverse set of development projects and watersheds representing a range of site characteristics, geographic locations, existing standards, and compliance approaches. The process

for identifying a representative sample involves identifying factors that drive the variable being estimated (e.g., location ), and designing and selecting a sample that includes the different possibilities

. Abt Associates’ analyses for this pro gram typically provide understanding of changes attributable to the policy or regulation relative to a baseline representing “business

- as

- usual” conditions in the absence of the policy or regulation. The experimental design therefore also involves defining the appropriat e analytic baseline considering existing conditions or reasonable projections over the period of analysis, including current industry trends, applicable regulations and standards, economic outlook, any applicable climate considerations, among others . Data gathering efforts may therefore include, in addition to the information above, assembling projections by government and industry trade associations (such as modeled electricity demand and supply or projected construction activities), and state and national regulatory requirements.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 15 To support the pro gram goals and achieve the specific project objectives specified in each work assignment under this contract, Abt Associates will g ather data from a variety of sources ( see d etail ed list

in Table 3

- 1

. The types of data needed for the environmental and economic regulatory analyses include,

but are not limited to:

 Characteristics of entities that may be affected by the policy or regulation (industry or public utilities) , including economic/financial data, ownership information, industry sector, number of employees, production levels, effluent quantities

, constituents

 Characteristics of waterbodies that the policy or regulation may affect , in cluding type, flow,

condition , water quality, land use within the watershed

 Baseline environmental conditions that may affect policy effectiveness, including land use, point and nonpoint sources, existing and projected levels of best management practice or treatment technology implementation

 Information about water pollution control technology implement ation, effectiveness, and costs

 Market conditions and trends in industries potentially affected by policies or regulations (e.g., demand, supply

, prices , com petition )

 Information about resources potentially affected by policies or regulations, including human health effects and incidence rates, non

- consumptive recreation (e.g., boating, swimming, wildlife viewing) activity levels, federally listed threatened a nd endangered species and location

 Information about value of use and non

- use benefits , including existence value of water quality improvements, commercial fishery value by species, cost

- of

- illness

 State and governm ent administrative expenditures

 Data on m acroeconomic conditions, wages, and interest rates.
Abt Associates anticipates that the sources of these data will include:

 Public literature and information available via the Internet

 EPA databases that the Agency designed and developed for the particular data need for this

pro gram

 Other EPA offices that have collected data from the industry and sources of relevance to this

pro gram

 State, local and other federal agencies

 Other organizations that may be identified during the course of the data identificatio n and collection process, such as federal agencies in other countries that are responsible for information

on cross

- boundary pollution.
Abt Associates anticipates that data collection and identification will be an iterative process in which:

 Initially gat hered materials may suggest or point to other sources of existing data

 Initial searches yield excessive amounts of information that, upon further review, we determine to be either not directly relevant or of a quality that is less than desired

 Initial searches yield too little data and, therefore, we refine the searches to yield additional

information

 Initial searches yield data that contains gaps in information needed by EAD

. Abt Associates will evaluate data identified during the searches based on relevance, level of detail, and types of information covered, as described in Section 3

. After conducting initial searches and reviewing results of those searches, Abt Associates will work with the EAD WAM to refine search criteria, and conduct additional data gathering efforts necessary to identify relevant, useful information, and fill gaps

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 16 identified in useful data sources. We anticipate that data gathering will continue throughout the period of performance of this contract.

In some instances, Abt Associates may conduct primary data collection activities to support decision

making. This supp ort typically includes surveys of regulated entities to assemble detailed plant

- specific
data. Section 4 addresses primary data gathering.



### 2.4 Overall D ata Quality Objectives and Criteria

EAD has determined that the data Abt Associates will be collect ing and us ing in environmental and economic analyses to support the ELGs program represents influential scientific, financial or statistical

information. As such, the Agency’s Information Quality Guidelines (EPA, 2002) establish a standard of quality for information, based on objectivity, utility, an d integrity. Accordingly, this PQAPP describes
how Abt Associates collect s , manipulate s , and analyze s data consi stent with th is standard of quality , and provide s transparency regarding the data source, assumptions, methods, and statistical procedures

. The overall factors that Abt Associates uses in reviewing data for use in activities under this pro gram include tra nsparency, accuracy, completeness, internal consistency, reasonableness, representativeness, and comparability. Below, we discuss considerations with respect to each of the three types of analyses

.



### 2.4.1 Environmental Assessments
Figure 2

- 3 provides a n example schematic summary of Abt Associates’ process for conducting
environmental assessments in support of ELGs or other regulations . The diagram maps the pro gram phases (horizontal paths), key steps (boxes and arrow labels ), and information flow (arrows).

Figure 2

- 3 : Process for Conducting Environmental Assessment

.

The overall objective of the e nvironmental assessment is to characteriz e and quantify the direct and ancillary effects of policies or regulatory actions on surface waters and other media. This assessment

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 17 generally involves compiling and analyzing data to describe base line environmental conditions, and using models to evaluate how the policy or regulation may improve water quality, reduce human exposure to pollutants, and protect or enhance ecosystems. As shown by the arrow to the right of the flow diagram,

t he outputs from the environmental assessment also provide some of the inputs used to quantify and
monetize the benefits of the regulatory action (see Section 2.4.3 )

. Abt Associates generally relies on existing data to support environmental assessments. These data characterize existing environmental conditions (effluent discharges, baseline concentrations), ecosystem or human health effects, and/or support develo pment of models to assess changes in these conditions and effects due to the policy or regulation. We may obtain this d ata from the literature, environmental data sets, models, and other sources

. We describe the procedures we use to gather, manage, assess, and analyze secondary data in Section 3 of this P QAPP.
The following data quality obje ctives apply to existing data that Abt Associates uses for qualitative or quanti tative environmental assessments:

 Reliability and accuracy : Abt Associates evaluates data for use in qualitative environmental assessments for reliability based on the apparent quality of the underlying scientific work as evidenced by the written source (e.g., published articles

, databases , models ) . This evaluation involves verifying that the source identifies assumptions and methodologies, that sufficient
information is availab le to evaluate the variability and uncertainty (quantitative and qualitative) in the information or in the procedures, measures, methods, or models, and that the assumptions and methodologies are consistently applied throughout the analysis as reported in the source. See

Section 3.3 for details on applicable procedures.

 Consistency and comparability : Source d ata need to be consistent across geographical and temporal dimensions relevant to an analysis. For national

- level analyses, Abt Associates preferably selects data that are collected using consistent approaches across geographic locations (states, counties, hydrologic regions)

. Abt Associates verifies that data collected to support environmental assessments are internally consistent and comparable to other similar data. This

check involves confirming that the sum of individual elements equals reported totals , or comparing data sets and information to identify material differences. See Section 3.3 for details on applicable procedures.

 Complete ness and representative ness : Abt Associates evaluate s data for use in environmental assessment based on how well the conditions they describe represent the diversity of waterbodies, discharge sources, and pollutants addressed in the analysis. This evaluation includ es considering
the date when the information was collected

, and differences in explanatory variables that may contribute to changes in observed conditions (e.g., changes in land use since data were collected or a model developed). See Section 3.3 for details on applicable procedures.
The objectives above drive not only our selection of the data used in environmental assessments , but also
the methods w e employ to compile, manage, and analyze the data, as described throughout this PQAPP. For example, Abt Associates ensures accuracy of the information used in analyses by following data processing procedures that minimize the potential for transcription er ror , and by running validation
checks both within data set and against other sources

. See Section 3.3 for details on applicable procedures.



### 2.4.2 Economic /Financial Analyses

Abt Associates conducts the following evaluations of the economic impacts of the ELGs program :

 Assessment of total social cos ts and benefits;

 Economic impact analyses (impacts on producers, consumers, etc.);

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 18  Cost effectiveness analyses;

 Regulatory Flexibility Act (RFA) analyses;

 Unfunded Mandates Reform Act (UMRA) analyses; and

 Analyses to support executive orders:

 Executive Order 1 3563 : Improving Regulation and Regulatory Review

 Executive Order 12866: Regulatory Planning and Review

 Executive Order 12898: Federal Actions to Address Environmental Justice in Minority Populations and Low

- Income Populations

 Executive Order 13045: Protection of Children from Environmental Health Risks and Safety
Risks

 Executive Order 13132: Federalism

 Executive Order 13211: Actions Concerning Regulations That Significantly Affect Energy Supply, Distribution, or Use

Figure 2

- 3 provides a n example schematic summary of Abt Associates’ process for supporting EAD economic/financial analyses , including the program phases (horizontal paths), key steps ( boxes and text arrows), data flow (arrows). Much of the data we use for these analyses are existing data (see Section 3 ), although we also collect primary data to support the analyses through surveys, as described in Section 4 of this QAPP.

Figure 2

- 4 : Process for Conducting Economic/Financial Analyses

.

The following overall d ata quality objectives apply to data for economic /fi nancial analyses :

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 19  Accuracy and Reliability : Abt Associates evaluates existing data for use in economic/financial analyses for accuracy and reliability based on the primary purpose and scope of the data collection. For example, federal agencies devoted to the primary purpose of collecting and reporting economic data, including the Census Bureau, Bureau of Economic Analysis, and
Bureau of Labor Statistics

, meet federal data collection standards , and are among the most reliable, ac curate, and consistent data for the economic analyses performed under this contract

. Abt Associates uses the most current data available from these sources, and updates data to a common year using appropriate price or cost adjustment indices also available from these

sources. Abt Associates fully documents the source, date, and any important considerations for comparability or gaps that the source may describe . These data cover measures of population, income, employment, wages, costs, and prices, by locatio n, industry, and demographic characteristics. Abt Associates also utilizes data produced by comparable state

- level agencies as
best available. For data from all other sources (e.g., peer

- reviewed journal, government

- funded study, trade association study), Abt Associates evaluates the accuracy and reliability of the data,

and discusses potential data quality issues with the EPA WAM. Abt Associates subsequently documents the source, data

, and rationale for use of particular sources and values, methods, and

ev aluations of the results of all data used in the deliverable document. Abt Associates also discusses potential biases, limitations, and uncertainties associated with the use of existing data in a separate section of deliverable reports. Abt Associates may use alternative sources to illustrate

uncertainties.

 Transparency : T he accompanying documentation must identify the primary purpose, sources, and

methods for developing the values , and describe any exclusions (e.g., to prevent disclosure of

confidential in formation) or adjustments (e.g., to report values in dollars of a given year)

. Where data involve estimation, the source must describe estimation methods, uncertainty

- related issues, and the statistical uncertainty associated with the estimates.

 Reasonable ness : The methods u nderlying development of values must be st andard methods for the type of study. For example, technology costs could be based on modeling software that utilizes a work breakdown structure of individual component costs based on vendor quot es and industry sources (e.g., RS Means).

 Complete ness and representative ness : Abt Associates uses data from a variety of sources to supplement or update the government and the peer

- reviewed academic literature, such as the Engineering News

- Record constru ction cost indices, and the Agency’s own databases of dischargers and discharges (Integrated Compliance Information System

- National Pollutant Discharge Elimination System, ICIS

- NPDES). These sources provide the most complete and representative information for specific data elements. Abt Associates evaluate s the data using the

criteria specified above, and fully documents the source, date, and any inconsistencies with other
sources of similar information or across data from the source. Abt Associates discuss es any data
issues with the EPA WAM, and mitigation approaches, which may include primary data gathering to rectify the in consistency or fill a data gap.

As for environmental assessments, t he se objectives drive our selection of data, and the methods we

emp loy to compile, manage, and analyze the data. See Section 3.3 for details on applicable procedures.



### 2.4.3 Benefits Valuation

The overall objective of benefits valuation is to develop monetary values of the direct and ancillary effects of policies or regulatory actions on surface waters and other media. This analysis involves using outputs from the environmental assessment to identif y human health, ecological, and economic productivity endpoints for which economic values exist or can be developed based on existing data, and estimating monetary values of benefits of the regulatory action

.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 20 Some of the benefits we expect from the ELGs a nd other regulations will manifest in economic markets through changes in price, cost, or quantity of market

- valued activities. For benefits endpoints traded in markets, such as increased yields from commercial fisheries, we measure benefits by market pric es or market

- based factor pricing. We also use c ompetitive prices to measure avoided cost type of benefits. For example, reduced pollutant loadings to public water supplies may lower costs of drinking treatment. We
also use m arket prices to value direct me dical costs of illnesses associated with exposure to pollutants.

In other cases, benefits involve activities or sources of value that either do not involve economic markets or involve them only indirectly. We monetize t hese benefits, referred to as “nonmarket” benefits, based on a variety of market valuation approaches, including primary research methods (e.g., stated preference surveys)

, and benefit transfer from existing studies.
In general, a nalyses of benefits from ELGs include the following be nefit categories and approaches :
 Direct market use values (goods and products consumed or processed, such as commercial fish and shellfish)

, estimated using market

- based approaches

 Nonmarket values (use values, such as recreation, aesthetic; nonuse values, including existence, aesthetic, bequest) estimated using benefits transfer, revealed preference approaches (including travel cost, hedonic pricing models, risk

- wage approaches) or stated preference surveys

 Indirect use values (ecosystem services such as f lood control, regulation of water flow and supply, and nutrient retention)

, estimated using productivity and cost

- based approaches (e.g., effect on production; replacement costs; cost of providing substitute services; cost of avoided damages)

. Figure 2

- 5 presents a sche matic o f Abt Associate’s process for conducting benefit s analyses , using the specific example of the Steam Electric ELG, showing the linkage between the ecological endpoint s af fected by the regulation and the quantification and monetization of benefits

. (The category of impoundment

- related effects specific to the steam electric industry may not be broadly applicable to all ELGs that EPA is developing.) The figure also distinguis hes between the primary benefits, which are the environmental improvements that are the target of the rulemaking, and secondary benefits which ma y occur in conjunction with achie ving the primary benefits. It is appropriate to account for secondary
benefits in the regulatory analysis, although these benefits are not the primary objective of the regulatory

action.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 21 Figure 2

- 5 : Example Process for Conducting Benefits Analyses

.

Abt Associates follows the general recommendations and principles for conducting benefit analyses

outlined in EPA’s Guidelines for Preparing Econo mic Analyses (EPA 240

- R

- 10

- 001 ; December 2010 )

, including identifying the baseline, identifying potentially affected benefits categories, approp riate methods for estimating benefits, including conducting benefits transfers, and discounting future benefits

. To develop these analyses

, Abt Associates primarily relies on existing data. These data characterize benefitting populations, society's willing ness to pay (WTP) to avoid adverse health effects (e.g., cancer
and cardiovascular disease ), the cost of an illness estimates (including medical costs, lost income, and other expenses )

, WTP for improving ecosystem services and goods, consumer demand for market goods, production functions, replacement costs, cost of avoided damages, In addition, existing data can be used

to support development of models or benefit valuation functions to assess changes in the relevant ecological or human health endpoints

. We may obtain this data from the literature, economic data sets, models, and other sources (see Section
3 )

. In some (likely rare) instances, Abt Associates may collect primary data to support the analyses through surveys or focus groups (see Section 4 )

. W e may al so develop a traditional (stand alone) QAPP to cover new benefits method developments. With respect to overall data quality objectives for benefits analyses, the following apply :
 Accuracy and Reliability : Abt Associates utilizes data from a variety of studies that reflect methods and techniques accepted in the literature as capable of producing accura t e and reliable information on the benefits of water quality and other environmental improvements. These

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 22 studies provide information on market and nonma rket amenity values, and related information for applying the values (e.g., measure of environmental change being valued, affected resource characteristics, availability of substitute sites, study methods, and other data ). For amenity values,

Abt Associate s uses studies from the peer reviewed literature, and primary government studies subjected to peer review ; f or related information, data sources are federal agencies devoted to the primary purpose of collecti ng and reporting economic data

. As described und er economic and financial analysis, d ata from these sources meet federal data collection standards, and thus are among the most reliable, accurate, and consistent data for the economic analyses performed under

this contract. Abt Associates uses the most cu rrent data available from these sources, and updates data to a common year using the appropriate price or cost adjustment indices also available from these sources. Abt Associates fully documents the source, date, and any important considerations for compa rability or gaps that the source may describe. For data from the peer

- review literature

and government research , Abt Associates evaluates the accuracy and reliability of the data, and discusses potential data quality issues with the EPA WAM. Abt Associates subsequently documents the source, data , and rationale for use of particular sources and values, methods, and
evaluations of the results of all data used in the deliverable document. Abt Associates also discusses potential biases, limitations, and uncerta inties associated with the use of existing data in a separate section of deliverable reports. Abt Associates may use alternative sources to illustrate
uncertainties.

 Transparency : In order to use a source, the accompanying documentation must clearly identi fy the methods, data, and assumptions underlying the results.

 Reasonableness : Abt Associates evaluates the study results compared to results from other comparable studies for reasonableness . For example, benefit values may fall within the range of
results estimated by other studies using different techniques , confirming the results. For estimates that fall outside the range estimated by other studies, we provide explanation of why the results
make sense.

 Complete ness and representative ness : Abt Associates u ses studies that provide information on market and nonmarket, and uses and nonuse values, and representative of the environmental changes that the environmental assessment indica tes will result from the rule. We supplement or update the values using cost a nd price indices as appropriate, as described for economic and financial analyses . These sources provide the most complete and representative information for specific data elements. Abt Associates evaluates the data using the criteria specified above, and fully documents the source, date, and any inconsistencies with other sources of similar information or across data from the source. Abt Associates discusses any data issues with the EPA WAM, and mitigation approaches, which may include primary data gatheri ng to rectify the inconsistency or fill a data gap.

See Section 3.3 for details applicable procedures on data quality assessment and use

.



### 2.5 Training and Certification

The Abt Associates staff , subcontractors, and consultants for this pro gram have the requisite academic training and professional experience in the disciplines relevant to the areas they support : economics, finance, engineering, biology, chemistry, ecology, public policy, communications, and other areas

. The Abt Associates Project Manager works with the Abt Associates Work Assignment Manager to ensure that work plans for individual WAs include t he appropriate expertise and experience ; the work plans document the proposed staffing plan. If EAD require s specialized skills not provided by the Abt Associates Team, the Abt Associates Project Manager will work with EAD to identify and obtain access

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 23 to the talent. The Abt Associates Quality Assurance Manager ensures that a qualified PQA Advisor
supports each work assignment

. A bt Associates will ensure that a ll staff involved in supporting this program receive a copy of this

P QAPP

, and of any work assign ment

- specific QAPP for projects they support. The se staff will have received the requisite instructions in the QA/QC procedures, best practices, and tools.



### 2.6 Project

- Level Documents and Records
The Abt Associates Program Manager and WAM are responsible for ensuring that project documents and records conform to technical requirements and quality system requirements specified in the WA or technical direction by the EPA WAM. The Abt Associates WAM s ets up the overall file management structure for project files on the server, and instructs staff as to the proper organization of detailed files within that structure. We typically organize project file folders by task, analysis type or issue, an d phase ( e.g., proposed vs. final rule). Table 2

- 3 below provides an example of project folder organization. Abt Associates WAMs may modify the example to fit the particular project needs.

Table 2

- 3 : Example project file management structure.

Folder Example subfolders and content

00

- Background Detailed study

01

- Admin Work plan and budget Meeting notes

02

- QAPP Quality Assurance Project Plan ( QAPP )
Approval sheets Audit findings

03

- Survey Questionnaire Information Collection Request ( ICR ) Survey data Quality Assurance ( QA )

04

- Cost and Economic Impact Industry profile Analyses Report

05

- Benefits Literature review Analyses Report

06

- Technical Support Briefings

Preamble and rule text

07

- Public Comments Comment summaries Comment response document

For example, t his PQAPP resides in a subdirectory of the contract

- level directory on the Abt Associates

server and is accessible to all staff working on the project. The Abt Associates Project Manager is responsible for reviewing the PQAPP at least annually to ensure that the content continues to be applicable to the project over time , and notifying the WAMs and PQ As of any updates. The Work Assignment Managers then notify project staff of the updates. The Work Assignment Manager is also responsible for notifying project staff of any updates to WA

- specific QAPPs or SQAPPs. Updates to

the se documents reside in the pr oject directories noted above, using a file naming convention that
includes the date and version number ( xx.xx.xx _v#) to signify revisions.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 24 At the project completion (or at other milestones determined by the Abt Associates WAM, PQA, or Project Manager), w e archive and write

- protect the work assignment file folder to prevent any changes to project files. In particular, we inventory files supporting a rulemaking record (proposal or final), and
archive and/or write

- protect the files to prevent changes.



### 2.6.1 CBI

A bt Associates implements folder or file security measures as needed to protect sensitive project data.

The Confidential Business Information ( CBI ) plan for Contract EP

- C

- 13

- 039 (developed under this work assignment ) describes our procedures for handling CB I, including information to be included in the

record for a rulemaking

.



### 2.6.2 Deliverables
The Abt Associates Program Manager and WAM are also responsible for ensuring that records and documents accurately reflect the completed work, and for ensuring that the pr oject team archives files are upon completion of the project

. Abt Associates produces a variety of deliverables under this contract

, including :

 Draft and final reports ( e.g., economic analysis report, environm ental assessment report, benefits analysis repo rt)

 Memoranda or issue papers describing methodologies , data sources, and assumptions

 Literature review summaries and databases

 Briefing slides or summaries

 Electronic files containing primary and existing data, analytic calculations, and results

 Electronic files (or hard copies if electronic versions are not available) of published literature,

we b pages, and other data sources

 Computer code and analytic framework , including programming comments and embedded

metadata

. The work plans for each work a ssignment identify the deliverables to be produced, and format (e.g., report, spreadsheet, database) for each deliverable. The Abt Associates WAM works with the EPA WAM

to identif y the appropriate level of review for deliverables considering the type of do cument, audience, project phase, and sensitivity or complexity of content. For most projects, written reports

– memoranda, issue papers, or formal reports

– are a primary way to communicate the results of an analysis to EPA and to the public. As such, Abt Associates follows QA procedures to ensure that such written reports support the project’s data quality objectives, and present the data, methodology, results, and limitations or sources of uncertainty accurately, clearly, and with details sufficient to al low the results to be reproduced.
When determining the QA review process of written documents, Abt Associates categorizes written reports into three groups, shown in Table 2

- 4 . Note that the QA review described in this table refers specifically to the written document, not the details of the content

, which will have been reviewed already as part of the underlying analysis or research task.

Table 2

- 4 : QA review categories for written deliverables

. Level Examples QA Review

1 Scoping memorandum

Summary table Briefing slides Outline review by WAM (and PQA Advisor if appropriate). Detailed review by author and peer (or WAM if appropriate ).
Numbers cited in the document are accurate, as verified in the underlying sources. Principal sources are cited. Principal limitations are noted.

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 25 Table 2

- 4 : QA review categories for written deliverables

. Level Examples QA Review
Document is free of grammatical errors.

Document uses established form atting style.

2 Draft analytic blueprint

Draft chapters for final report Level 1 plus : Detailed review by WAM for accuracy, clarity of presentation, and completeness.

All sources are cited.

Limitations and biases are described. Optional review by editor. Optional formatting review.

3 Economic Analysis report Level 2 plus : Editorial and formatting review (clarity of presentation, consistency of formatting, etc.). Final acceptance by Project Manager or his designee, including verifying use of appropriate l anguage and
characterization of the results.

PQA = Project Quality Assurance QA=Quality Assurance WAM = Work Assignment Manager

The Abt Associates WAM reviews all written deliverables (memoranda, reports, issue papers, etc.), as

approp riate, and identifies a qualified reviewer for material he/she authored. The reviewer assesses the clarity and accuracy of the written document, including verifying that the document has a clearly identified purpose; the findings are clear and supported by the inform ation presented; and that any analysis results and data match information from the underlying analytic files (spreadsheets, program outputs, etc.).
For longer reports targeted at the public and non

- EAD audiences, a skilled technical editor conducts a
final editorial review. The editor’s role is to verify correct grammar, spelling, and text organization/flow, as well as to ensure that the style and flow of the document are appropriate for the intended audience. Upon direction by the EPA WAM, Abt Associates facilitates external peer review of the methodologies and finished products.
After Abt Associates’ internal technical reviews have been completed (and the EPA WAM’s reviews as well, as appropriate), the relevant project management staf f

– usually the individual content area

managers

– address all comments and revisions. For major deliverables and final reports, such as reports
that will be part of the rulemaking record, the Abt Associates Project Manager or designee will provide an

addi tional final acceptance before submittal to EPA

. Individual reviewers complete the applicable signature form s upon completion of the requisite review items. Appendix A provides the deliverable review and signature forms /checklists that we will use for this contract.
Throughout the performance of the work, Abt Associates project staff maintain a log of substantive team discussions about the analyses, data issues, and decisions that affect the project. We also use t his log (or

differ ent logs for specific asp ects of a project) to document the outcome of discussions with the EPA
WAM that result in modifications to the analysis, and note any necessary follow

- up s

.



### 2.6.3 Quality Audits
On a quarterly basis, the Abt Associates Project Manager (o r his designee) will review implementation of this PQAPP under select work assignment s. This QA review/audit

, which will involve the WAM, PQA
Advisor , and project staff if applicable, will include discussion of ongoing QA/QC procedures,

### Project Management

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 26 documentation of su ch procedures, and discussion of any issues encountered in the performance of the
work

, along with any mitigation measures, needed adjustments to QA/QC procedures, or other actions to be implemented by the project team. The Project Manager (or his designee) will record performance and outcome of the quarterly review using the form depicted in Appendix B . The WAM will inform the EPA WAM and/or Project Manager of the status of any follow

- up items identified dur ing the meeting.

The WAM wi ll document QA/QC procedures, and any adjustments to the procedures, i n the monthly progress reports, as described in Section 3.3.5

. The Project Manager will address any identified internal needs, such as training, through means provided by the company (e.g., professional development plans for staff, brown bag s eminars).

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 27 

### 3. Existing Data

This section describes Abt Associates ’ procedures for data gathering, management, QA

, and use pertaining to existing data

.



### 3.1 Data Gathering

Abt Associates ’ support to EAD for the ELG progra m primarily
involves use of existing data to answer analytical questions . We

may use existing data exclusively, or in combination with primary data, to provide timely and effective basis for conducting the range of analyses under this project. Since we oft en obtain these data from publicly available sources , existing data may also provide transparency to the analyses , which is one of
the overall data quality objectives discussed in Section 2.4

. Abt Associates works with the EPA WAM to define data needs for the work assignment , considering the
intended use of the data. EPA defines these needs in the work assignment PWS; subsequentl y, Abt Associates discusses the needs in the technical approach described in our work plan, meetings and discussions, and planning memoranda detailing our understanding of the analytic issues and potential approaches.
The following sections describe our p rocess for gathering data, differentiating across the three types of analyses where appropriate.



### 3.1.1 Data Needs
Table 3

- 1 provides a detailed description of common d ata needs for the three types of analyses conducted
under this contract , and sources for those data , consistent with the overall objectives discussed in Section

2.4

, and based on ongoing and recent analyses performed for EAD

. For each data so urce, we also summarize the rationale for selecting the data, consistent with the criteria discussed in Section 3.1.3

. Table 3

- 1 summarizes the publicly available existing data sources that we use in our analyses . Abt Associates also uses data generated by EPA or its other contractors to support a specific rulemaking

. Examples of such data include outputs from engineering analyses (technology costs, loading reductions, etc.) that we use as inputs in our economic impacts, environmental assessment, or benefits analyses. We

manage these data similarly to “secondary data ” i n the sense that they are not data we obtained or generated internally. However, we do not apply the same level of applicability review as other obtained secondary data because EAD has already deemed the quality of these data to be acceptable through

compa rable PQAPPs, SQAPPs, or QAPPs associated with the contracts under which such data is

obtained

. Existing Data Sources

 Literature

 Published databases

 Existing models

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 28 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

Land Cover and Land Use Data

U.S. Geological Survey ( USGS ) Hydrologic Unit
Code (HUC) geospatial
data Watershed hydrologic
resources and boundaries Estimating the effect of regulatory options; identifying affected sources of pollution Federal government source. Standard geographical unit
definitions used across federal government agencies. http://water.usgs.gov/GIS/h
uc.html

Environmental Protection Agency ( EPA ) NHDPlus geospatial data G eo

- spatial, hydrologic
framework dataset Modeling the effect of regulatory options Federal government source. Standard geographical unit
definitions used across federal government agencies. Publicly available and widely used. http://www.horizon

systems.com/nhdplus/

Google Earth Desktop mapping software that shows current and (where available) recent historical satellite images R eviewing, viewing, and collecting spatial data on land and water features; quickly viewing land

- use a nd land

cover change over time Global maps and spatial data which are publicly available, widely used, and free. Recent images are 1

- 3 years old and updated periodically. Images have up to 1

- meter resolution in many areas. Can be used to develop shapefi les for analysis in GIS software. http://www.google.com/eart h/explore/products/desktop.
html

Integrated Climate and
Land Use Scenarios (ICLUS) Data associated with s cenarios broadly consistent with global

scale, peer

- reviewed storylines of population growth and economic development, which are used by climate change modelers to develop projections of future climate Projecting future land use and impervious surface cover based on assumptions about population growth, transportation patterns Federal gover nment source.

National

- level data developed using consistent methodology, agreed upon assumptions. Publicly available. http://www.epa.gov/ncea/gl
obal/iclus/

National Land Cover Data (NLCD) Consistent land cover data applied across the
conterminous United States at a spatial resolution of 30

meters Identifying land types and potential pollutant sources Federal government source. Most comprehensive national

- level
dataset. Publicly available and widely used. http://www.mrlc.gov/nlcd20 06.php

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 29 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

U.S. Fore st Service
Urban Forest Data and i

Tree Data Databases and publications, including city

- specific analyses, addressing urban forest services including rates of tree canopy cover, human health, environmental quality and energy savings Estimating ancillary be nefits of reductions in impervious services (i.e., increases in

green space) Federal government source. Publicly available and widely used.

National

- level datasets. http://www.nrs.fs.fed.us/dat
a/urban/ ; http://www.itreetools.org/

Ecological and Health Risk Data

F ish and Wildlife Service (F WS ) Threatened and Endangered Species System database Provides a list of species designated as threatened or endangered and designated
critical habitat areas Identifying threatened and endangered species that may be affected by regulatory options Federal government source. Most comprehensive national

- level
dataset for data. Publicly available. http://ecos.fws.gov/ecos/ho
me.action

EPA Integrated Risk Information System (IRIS) I nformation on health effects from exposure to
environmental contaminants Identifying endpoints, risk values and key studies for estimating baseline human health risks, and risk reductions from regulatory options; key
values include unit risk estimates, or regression equations from primary studies Federal government source.

National

- level dataset collected using consistent methodology over
time and across industries. Subject to extensive scientific / QA review prior to public ation. Publicly available and widely used. http://www.epa.gov/iris/

Agency for Toxic
Substances and Disease Registry (ATSDR)
Database Toxicological p rofile for pollutants Identifying endpoints, risk values and key studies for estimating baseline human
he alth risks, and risk reductions from regulatory options; key
values include unit risk estimates, or regression equations from primary studies Federal government source.

Peer

- reviewed summary of the toxicological and adverse human
health information on sev eral contaminants. Publically available. http://www.atsdr.cdc.gov/to xprofiles/index.asp

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 30 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

National Toxicology Program (NTP) Frequently updated Report on the Carcinogens

Reviews of adverse health
effects for other contaminants
in NTP Monographs Identifying endpoints, risk values and key studies for estimating baseline human health risks, and risk reductions from regulatory options; key
values include unit risk estimates, or regression equations from primary studies Federal government source. Systematic clas sification of the Weight

- of

- Evidence for carcinogenicity NTP Monographs are detailed, peer reviewed summaries of the
evidence for adverse health effects http://ntp.niehs.nih.gov/ntp/ roc/twelfth/roc12.pdf ; http://ntp.niehs.nih.gov/?obj
ectid=497C419D

- E834

- 6B35

- 8AF15D389859AF07

California EPA Provides several peer reviewed risk values; may be updated more recently than
IRIS or other EPA sources Supplementing risk values for endpoints identified through IRIS, ATSDR or NTP; key
values include unit risk
estima tes, or regression equations from primary studies State government source. Scientifically sound, peer reviewed
risk estimates http://oehha.ca.gov/about.h
tml

International Agency for
Research on Cancer (IARC) Provides peer reviewed weight

- of

- evidence
clas sifications for carcinogens Supplementing information from IRIS, ATSDR and NTP Scientifically sound, peer reviewed, internationally known weight

- of

evidence classifications http://monographs.iarc.fr/E
NG/Classification/

Water Quality and Discharge Data

EPA water quality criteria
and maximum contaminant level (MCL) Ambient and drinking water
levels of constituents protective of aquatic life, human health, and aquatic

dependent wildlife Estimating environmental benefits of regulatory options Federal govern ment source. Regulatory/statutory values. Publicly available. http://water.epa.gov/scitech /swguidance/standards/crit eria/current/index.cfm; http://water.epa.gov/drink/c
ontaminants/

USGS National Water Information System (NWIS) W ater resources data from approximately 1.5 million sites
nationwide Assessing baseline pollutant
concentrations and benefits of regulatory options Federal government source.

National

- level dataset collected using consistent methodology over
time and across industries. Publicly a vailable and widely used. http://waterdata.usgs.gov/n
wis

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 31 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

EPA’s Watershed Assessment, Tracking &
Environmental Results System (WATERS) Water quality information including designated uses, water quality assessment and monitoring results, causes and

sources of impairments, public beach closures, and discharger locations Assessing baseline
environmental conditions and benefits of regulatory options Federal government source. Most comprehensive source of information on water quality

assessments. Incorporates verified information from other sources (states). Publicly available. http://www.epa.gov/waters/i
r/

EPA’s Safe Drinking
Water Information System (SDWIS) Information on drinking water systems, including characteristics, monitoring results, violations, and
enforcement actions Identifying environmental risks to drinking water systems, and benefits of regulatory options Federal government source. Comprehensiv e national

- level source for drinking water treatment systems. http://water.epa.gov/scitech
/datait/databases/drink/sdw
isfed/

EPA’s Integrated Compliance Information System

- National Pollutant Discharge Elimination System (ICIS

- NPDES) 29 Information on permit ted discharges to surface waters Characterizing the types and quantities of pollutants discharged to waters by NPDES permittees; assessing compliance with permitted limits Federal government source. Most comprehensive national

- level dataset for a NPDES per mits. Publicly available and widely used. http://cfpub.epa.gov/npdes/ npdesnoi.cfm

USGS Reservoir Sedimentation Survey Information System (RESIS) The most comprehensive compilation of data from
reservoir sedimentation surveys throughout the
conterminous United States Evaluating baseline reservoir
sedimentation rates Federal government source.

National

- level datasets. Publicly available. Most compr ehensive source of reservoir sedimentation survey
data. http://pubs.usgs.gov/ds/ds4
34/

Air Quality and Emissions Data

Meteorological Data from EPA’s Support Center for Regulatory Atmospheric Modeling (SCRAM) Surface and upper air meteorological data from
stations across the United
States Estimating mixing heights for converting mass of air pollutant removal to changes in
concentrations Federal government source.

Used in the relevant literature. Publicly available. http://www.epa.gov/ttn/scra m/metobsdata_databases.
htm

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 32 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

U.S. EPA Emissions and
Generation Resource Integrated Database (eGRID) Database of environmental characteristics of power generation in the U.S. including air emissions, emissions rates, net generation, resource mix, and
other attributes Estimating the benefits of changes in energy usage under regulatory options Federal government source. Publicly available and widely used.

National

- le vel datasets. http://www.epa.gov/cleanen ergy/energy

resources/egrid/index.html

Company Financial Data

Financial Reports to the
US Securities and Exchange Commission Qualitative and Quantitative information on the reporting company’s financial condition Characterizing the financial condition of a company and/or industry category Federal government source. Official reports to meet statutory regulatory requirements http://www.sec.gov/about/a nnrep.shtml

Federal Energy Regulatory Commission ( FERC ) Form 1 /1

- F data Comprehensive financial and operating report submitted for Electric Rate regulation and
financial audits Characterizi ng the financial an d operating conditions of electric
utilities Federal government source. Official reports to meet statutory regulat ory requirements http://www.ferc.gov/docs

filing/forms.asp#1F

Value Line Investment Analyzer Proprietary financial data for over 1,000 publicly traded
firms Characterizing the financial condition of a company and/or industry category Contains information for 250 data fields, on approximately 1,700 major stocks, representing nearly
100 industries. 10 years of data history. http://www.valuelinepro.co m/Services/Equity/The

Value

- Line

- Investment

Analyzer/

Dunn & Bradstreet Company

- level data including
NAICS co de, employment, location, revenue and ownership type Characterizing the financial condition of a company and/or industry category and characterizing baseline financial condition for use in analyses L argest commercial database compiled thousands of sources

. Subject to extensive and rigorous QA procedures. http://www.dnb.co m

EPA’s Electronic Notice of Intent (eNOI) Information on sites/facilities that apply for coverage under selected general permits ( Construction General Permit (CGP), Multi

- Sector General
P ermit (MSGP), Pesticides General Permit (PGP) and Vessel General Permit (VGP) ) Characterizing facilities potentially subject to regulatory options Federal government source.

National

- level dataset for a various types of general permits. Publicly available and widely used. http://cfpub.epa.gov/npdes/ npdesnoi.cfm

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 33 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

Industry Financial Data

US Census Bureau
Economic Census Census of economic and employment data by NAICS and size categories, conducted every 5 years for all industrial sectors except the agricult ural and mining sectors Estimating the number of entities in an industry category; characterizing baseline
economic/financial conditions (employment, revenue, trends) Federal government source.

National

- level dataset collected using consistent methodology over time and across economic sectors. Subject to extensive QA review prior to publication. Publicly available and widely used. http://www.census.gov/eco
n/

US Census Bureau Quarterly Financial Report Quarterly financial data for manufacturing, mining , trade
and selected service industries Characterizing the financial performance and condition of an industry category Federal government source. Official government source for the
data. Publicly available and widely used. http://www.census.gov/eco n/qfr/

B oard of Governors of
the Federal Reserve System, Economic &
Research Data Industrial production and capacity utilization data by selected industry sectors Characterizing the financial condition of an industry category Federal government source. Official government source for the
data. Publicly available and widely used. http://www.federalreserve.g
ov/econresdata/default.htm

US International Trade Commission, Interactive
Tariff and Trade
Dataweb 6 Data on US tariffs and
international trade Charact erizing import competition and export dependence for an industry category Federal government source.

International

- level dataset collected using consistent methodology over
time and across industrial sectors. Publicly available and widely used http://dataw eb.usitc.gov/

The Risk Management Association, eStatement
Studies Proprietary financial data by
NAICS code Characterizing the financial condition of an industry category D ata comes directly from the
financial statements of institutions. 15 years of data h istory http://www.rmahq.org/tools

publications/tools/estateme

nt

- studies

Standard & Poor’s Industry Surveys and Stock Reports Financial information for selected companies and overall industry, including rating values (e.g. debt rating) and industry outlooks Characterizing the financial performance and condition of an industry category Tracks data for 50 North American
industries and 10 Global industries. Trend analysis by top economists. http://www.netadvantage.st andardandpoors.com/

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 34 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

Other Industry Data

Small Business
Association (SBA) Business size standards, by industry sector Identifying small entities Federal government source. Regulatory/statutory values. Publicly available. http://www.sba.gov/categor y/navigation

structure/contracting/contra cting

- officials/eligibility

- size

standards

US Census Bureau
Census of Governments Census of all state and local government organiza tion units
conducted every 5 years C haracterizing baseline public finance and employment
conditions ; comparing compliance costs to receipts Federal government source.

National

- level dataset collected using consistent methodology over time and across jurisd ictions. Subject to extensive QA review prior to publication. Publicly available and widely used. http://www.census.gov/gov
s/

U.S. Energy Information Administration Electricity
Data Browser Data on the generation, sale, and consumption of electricity Char acterizing baseline operations, and
economic/financial conditions of the electric power sector Federal government source.

National

- level dataset collected using consistent methodology over
time and across industries. Subject to extensive QA review prior to publication. Publicly available and widely used. http://www.eia.gov/electricit y/data/browser/

U.S. Energy Information
Administration Annual Energy Outlook Yearly projections and analysis of energy topics Supporting analyses of economic impacts of regulat ory options (e.g., estimating future energy demand and cost) and benefits (e.g., estimating benefits of energy savings) Federal government source. Meets specific data needs

Based on best available data. Publicly available. http://www.eia.gov/forecast
s/aeo/

U.S. Energy Information Administration, Other
Data sets Various data sets on the production, prices, disposition of energy (electricity, coal, natural gas, petroleum, renewables, etc.), and energy consumption Characterizing baseline operations, and
economic/financial conditions of the energy sector Federal government source. Meets specific data needs. Publicly available. http://www.eia.gov/

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 35 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

USGS Minerals
Yearbook Annual statistics on material
and mineral industries (production, consumption, etc.), economic and technical
trends

, and development Characterizing baseline operations, and
economic/financial conditions of
the material and mineral
industries Federal government source.

National

- level dataset collected using consistent methodology over
time and across industries. Subject to extensive QA review prior to publication. Publicly available and widely used. http://minerals.usgs.gov/mi nerals/pubs/myb.html

USGS Minerals Commodity Summaries Annu al statistics on minerals
and materials domestic industry infrastructure, government programs, tariffs
and statistics Characterizing baseline operations, and
economic/financial conditions of
the material and mineral
industries Federal government source.

Na tional

- level dataset collected using consistent methodology over
time and across industries. Subject to extensive QA review prior to publication. Publicly available and widely used. http://minerals.usgs.gov/mi nerals/pubs/mcs/

Grubb & Ellis, Inc. Reed
Cons truction Data (RSMeans) Information on the
characteristics of construction projects (e.g., size and value) Characterizing commercial/
industrial and residential construction projects; estimating the economic impact of regulatory options Accurate and up

- to

- date cost
information for thousands of items. http://rsmeans.reedconstru
ctiondata.com/

IHS Economics and Country Risk
Construction Market Analysis and Forecasts Construction market analysis
and forecast Characterizing baseline operations, and

econo mic/financial conditions of
the construction sector Insight from leading macroeconomic experts on the construction industry

. Covers U.S. and global markets.

Based on data from U.S. Census
Bureau. http://www.ihs.com/product s/global

insight/industry/construc tio n/index.aspx

National Oceanic and Atmospheric Association (NOAA) Commercial Landings Statistics Annual and monthly data on
commercial fish landings (in metric tons/pounds and dollars) by species and geographical area Estimating commercial fishing
value s Federal government source.

National

- level dataset collected using consistent methodology over
time. Clear description of data caveats
available. Publicly available. http://www.st.nmfs.noaa.go
v/st1/commercial/

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 36 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

NOAA’s National Marine
Fisheries Service (NMFS) Marine Recreational Fishing Statistics Survey N early 30 years of data on marine recreational catch, effort, and participation throughout the w orld E stimating the impact of recreational fishing on marine

res ources Federal government source.

National

- level dataset collected using consistent methodology over
time.

S tandardized data

. S cientifically credible and
consistent data

. Publicly available. http://www.sefsc.noaa.gov/
about/mrfss.htm

Property Transaction
Records and
Characteristics Files State

- level databases (Maryland, New York, and others) of real estate transaction records, also including state

- level aggregations of property
assessment data on structural and parcel characteristics Conducting hedonic property value analyses of the value of

env ironmental amenities and other parcel

- level analyses ( commercial or industrial analyses; sprawl analysis ) State government source.

Need for state

- specific data based on current models (New York, Maryland). Data are aggregated from standard property transf er and assessment
forms submitted at local jurisdictions. Subject to QA review by state, county, and/or municipal agencies. Availability depends on specific data source. For example, NY data are free with a FOIL request. MD
data are not free. http://www.tax.ny.gov/resea rch/property/online/intro.ht

m http://swcf.tax.ny.gov/sales
web/ http://planning.maryland.go v/OurProducts/Prope rtyMa pProducts/MDPropertyView
Products.shtml

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 37 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

Parcel Map Layers GIS dataset of point features for real property tax parcels by county or state (New York, Maryland, and others). These points represent the visual center of each parcel on a tax

map Conducting hedonic property value analyses of the value of
environmental amenities and other parcel

- level analyses State government source.

Need for state

- specific data based on current models (New York, Maryland).

State

- level dataset of parcel point layers, including map layer versions from recent history. Can be readily merged with other data, including other GIS layers and property transaction records
from the relevant state. Availability depends on specific data source. For example, NY data
are free with a FOIL request. MD
data are not free. http://gis.ny.gov/gisdata/inv
entories/results.cfm?themeI
Ds=2 http://planning.maryland.go v/OurProducts/PropertyMa pProducts/MDPropertyView
Produc ts.shtml

Population and Demographic Data

US Census Bureau
Decennial Census , State & County QuickFacts,
and American Community Survey Census of population and
socioeconomic data conducted every ten years, and ongoi ng statistical survey that samples

a percentage of the population

every year Modeling and analysis, including environmental justice, of population, housing, and
other socio

- demographic characteristics; benefits
valuation functions Federal government source.

National

- level dataset collected

us ing consistent methodology over
time and across economic sectors. Subject to extensive QA review prior to publication. Publicly available and widely used. http://www.census.gov/201
0census/ ; http://quickfacts.census.gov /qfd/index.html# ; https://www.census.gov/ac
s/www/

Woods and Poole
Economics Inc. Database with projected economic and demographic data at the state, region, county, and metropolitan area levels through 2040 Adjusting future populations to account for population changes in benefits analyses Meets specific data needs.

National

- level dataset. Updated annually. http://www.woodsandpoole.

c om/

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 38 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

Valuation and Economic Impact

Literature on the Value of Improving Aquatic and Terrestrial Ecosystem
Services and Goods Studies in the literature that

c ontain estimates of willingness to pay (WTP) for surface water quality improvements, reduced risk to aquatic, terrestrial and avian species, changes in open and vegetation Developing a WTP value or a
valuation function See Section 3.1.3 See Appendix C

Interagency Working Group on Social Cost of
Carbon Report by the U.S. government interagency working group on
social cost of carbon presenting values by year under various discount rates Estimating the benefits of changes in greenhouse gas emissions under regulatory options Federal government source. Publicly available and widely used. Interagency report. http://www.whitehouse.gov/
sites/default/files/omb/ infor eg/social_cost_of_carbon_f or_ria_2013_update.pdf

FWS National Survey of Fishing, Hunting, and
Wildlife

- Associated
Recreation Data on participation in hunting, fishing, and wildlife viewing by region;

Q uantif ication of the economic impact of wildlife

- based
recreation Estimating economic impacts and benefits of regulation affecting wildlife

- based
recreation Federal government source.

National

- level and state

- level data. Data collected by U.S. Census Bureau computer

- generated interviews (standardized). Publicly available. http://wsfrprograms.fws.gov /Subpages/NationalSurvey/ National_Survey.htm

Practice Data

International Stormwater Best Management
Practice ( BMP ) Database D atabase of over 530 BMP studies, performance analysis results, tools for use in BMP performance studies, monitoring guidance and other study

- related publications Evaluating performance of modeled stormwater BMPs (i.e., validate results obtained using
the Soil and Water Assessment
Tool ( SWAT ) or other models) Sponsored by EPA and indus try

groups.
Most

- comprehensive and most

cited database of BMP information. http://www.bmpdatabase.or g/

U .S. Army Corp of Engineers (U SACE ) Dredging Information System Database of USACE and USACE contracted dredging jobs include volume dredged, data, location, and cost Estimating benefits of reduced
sedimentation and reduced dredging under regulatory options Federal government source.

National

- level datasets. Publicly available.

Detailed source for both volumes
and costs. http://www.navigationdatac

ente r.us/dredge/dredge.ht

m

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 39 Table 3

- 1 : Detailed Description of Existing Data

. Data Source Description Typical Data Need/ Use Criteria for Selecting Resource

Stormwater Management
Manuals State

- specific manuals that specific design criteria for various stormwater practices Estimating the amount of each vegetation type in “acres of vegetation per acre of BMP” Need for state

- specific data. Publicly available. Varies based on state.

Indices

Bureau of Economic Analysis GDP deflator Index of inflation Updating costs and benefits to common year values Official federal government source
for the data. Publicly available and widely used. http://www.bea.gov/

Bureau of Labor Statistics Analysis
Consumer Price Index (CPI) Monthly data on price changes for a representative basket of goods and services, purchased by urban

consumers Updating costs and benefits to common year values Official fed eral government source
for the data. Publicly available and widely used. http://www.bls.gov/cpi/

E ngineering N ews

R ecord Construction
Cost Index (CCI) Construction cost index Updating construction

- and engineering operations

- related costs/prices to common year values Based on actual prices reported from the same suppliers in 20 U.S.
cities Widely used. http://enr.construction.com/
economics/

B ureau of L abor
S tatistics, Employment
C ost I ndex and wages (ECI) Inflation index for wages and benefits by region and sector Updating components of costs/prices (except
construction and non

- labor operating costs) to common year values; estimating labor
costs associated with burden
hours Federal government source.

National

- level dataset collected using consisten t methodology over
time and across industries. Subject to extensive QA review prior to publication. Publicly available and widely used. http://www.bls.gov/data/

Bureau of Labor Statistics, Producer Price
Index Index of the average change in output prices received by domestic producers Characterizing product pricing in an industry category Federal government source.

National

- level dataset collected using consistent methodology over
time and across industries. Subject to extensive QA review prior to publication. Publicly available and widely used. http://www.bls.gov/ppi/

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 40 

### 3.1.2 Potential Data Sources

Abt Associates relies on the existing data listed in Table 3

- 1 and other data generated specifically for the analysis by EPA or other contractors (under separate QAPPs) to characterize economic or environ mental conditions that may change because of the regulatory options (and support evaluation of the changes), and to provide key values for analytic frameworks. If we determine that these data are not sufficient to support the analyses, we will identify add itional data sources and describe these sources in a supplemental QAPP or amend this PQAPP. We describe the process for identifying potential data sources below.

Sources of additional existing data include academic journals and other peer

- reviewed publicat ions, government reports and databases, and working papers. Abt Associates will identify additional data sources through online searches (see Literature section below for discussion of search tools and key words) , and bibliographies or publications from government organizations (see below for a list of sources)

. Abt Associates compiles data used on prior projects done for EAD and other clients

, and continuously review s relevant literature on economic and environmental data to support the environmental and economic regulatory analyses under this pro gram

. For example, Abt Associates will use its experience from prior regulatory impact analyses and environmental assessments, Internet searches, and personal communications to identify estimates of unit costs and benefits from water pollution controls and water quality improvements specific to a given industry.

We discuss our approach for conducting research of existing literature

, published databases , and models
below.

Literature

Abt Associates uses searches of the following to identify relevant literature :
 G eneral literature databases and search engines (EBSCO host , Google Scholar, Google)

 T opic

- specific re ference and abstract databases such as

 benefit valuation literature : Environmental Valuation Resource Inventory (EVRI), Benefits Use Valuation Database (BU VD), AgEcon Search, EconLit, and RePEc/IDEAs

,

 Soil and Water Assessment Tool (SWAT ) database co mpiled by Texas A&M University

,

 health effects; Pub M ed and T OXNET

 W ebpages of authors and university programs known to publish relevant research (e.g., on state d preference studies and/or water quality valuation )

 W eb sites of organizations and agencies known to conduct relevant research (e.g., Resource for the Future, EPA National Center for Environmental Economics for research on resource economics valuation; U.S. Geological Survey ( USGS ) and EPA Office of Research and Development for research on water quality impacts of pollutants ).

 K ey resource economics journals (Land Economics, Environmental and Resource Economics, Marine Resource Economics, Journal of Env ironmental Economics and Management, Water

Resources Rese arch, Ecological Economics, Review of Environmental Economics and Policy, and Journal of the Association of Environmental and Resource Economists).
When conducting literature reviews, we identify pot entially relevant articles and reports based on title, keywords, and abstracts

, either by direct ly search ing electronic publication databases or websites or by reviewing the literature cited in other relevant sources. We identify search terms from a prelim inary list of relevant topics for the particular analysis or type of impact. Based on results of the searches, we then either expand the terms to be more general or refine them to get more specific entries (e.g., searches for “stormwater AND runoff AND imp acts” may be refined to focus on “stream erosion” instead of general

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 41 impacts ; ensuring the correct chemical form such as Hg versus MeHg ). For example, we may identify other terms often associated with the desired keywords from articles that seem relevant u pon initial
review.
We then review the abstracts to screen relevant articles or reports and, if warranted by the breadth of the research and number of articles, categorize the articles and reports according to topics and subtopics. To the extent possible, we prioritize literature from peer

- reviewed journals and government reports. We then carefully review sources that pass this screening step, and summarize and synthesize information in a
short write

- up or tabular format. We review the documented methods fo r rigor of study design, methods,

and results.
Abt Associates staff identifying and reviewing the literature have training in the disciplines pertaining to the research topics

– economics, engineering, environmental sciences

– and are qualified to assess the relevance of the data, quality of the analysis, and appropriateness of the results and conclusions against the objectives

. I f we reject a relevant article as a result of the review, we document the reason for this rejection in the accompanying notes

. For example, the following describes the literature search p rocess for recent ELGs.

 Step 1: We work with the EPA WAM to identify the issues to be covered by the review. This step may include developing a method memo randum that includes the proposed outline for the eventual report that will summarize the findings of the review, and identif ying the search parameters (search tools, period of interest, search terms, review/reduction process). Appendix C

lists the search terms we are using for two meta

- analys e s

.  Step 2: We accumulate abstracts, papers, and data sources in a reference database for more
careful review.

 Step 3: We periodically r eview interim findings with the EPA WAM, noting progress with identifying sources for the various outline topics and apparent gaps.

 Step 4: We review and categorize sources in accordance to relevance to the outline topics. We also include metadata, and dat a to help assess the quality of the sources : EndNote DCN ; Reviewer and Date ; Authors ; Year ; Title ; Additional Citation Information ; Length ; Resource type (Journal Article/Report/Information Request/etc.) ; Source (Government, Federal/Government,
State/Acade mic/Industry/etc.) ; Peer Reviewed (Yes/No/Uncertain) ; Bias (None/Unclear/Anti

SGE/Pro

- SGE) ; Not Relevant? (Flag) ; Notes ; Internal Notes ; Flag for relevant topic s (based on outline)

.  Step 5: We summarize findings or data from each of the papers relevant to a particular outline topic. A second staff reviews the summary of selected findings (number determined by the WAM) to ensure that the findings are characterized correctly.

 Step 6: We compile the findings into a narrative that summarize the overall findings of the literature review according to the outline.

Data Sets Other sources of existing data include published databases, data sets that accompany a given study, etc. as identified typically through Internet searches , government websites, expert suggestions

, or published
studies identified above. Considerations for identifying potentially relevant existing data bases are similar to those used to evaluate published literature. The initial review of the source includes assessing whether
and how the o riginal purpose of the data collection or underlying study may affect usability of the data for a different purpose . In addition, the initial review would consider whether other factors such as data year,

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 42 originating agency, collection method, calibration and validation procedure, stringency of the review and
comment process, etc.

, fit the data needs agreed to with EPA.

Models

Abt Associates uses models to simulate the effect of the policy or regulation on environmental conditions (for environmental assessment or benefits) or affected entities (for cost and economic impacts analyses). In selecting models to support analysis of a rule, we review exi sting models for their applicability to simulate the anticipated effects of a regulation

– based on the industries, pollutants, affected media,

processes, ability to specify parameters to represent the policy, and estimate policy

- relevant endpoints etc.
To provide greater transparency of the result and facilitate independent, public review and comment on the analyses, we preferably identify models available in the public domain over proprietary models.
Sources of information about existing models include p ast EPA rulemaking analyses, list of EPA

supported computational models (e.g., models listed on EPA ’s OST modeling portal ) 1

, and other Federal government agencies such as the U.S. Forest Service and the U.S. Geological Service

. Table 3

- 2 lists the models Abt Associates uses /used to support current /recent rulemaking analyses . We also provide the rationale for selecting these models, based on the criteria discussed in Section 3.1.3

.

1 The portal (http://www.epa.gov/epahome/models.htm) provides links to models, databases, and tools developed (including peer review) or supported by EPA offices, labs, and centers that EPA intends for use in regulatory analyses to support decision

- making.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 43 Table 3

- 2 : Models used to support regulatory analyses.

Data Source Purpose Criteria for Selecting Resource

SPARROW ( Spatially Referenced Regression on
Watershed Attributes ) Environmental Assessment Regression

- based modeling approach. Currently used for national modeling of the sources, fate and transport of nitrogen, phosphorus, and suspended
sediment in reaches of the coterminous United States (E2RF1 resolution). Federal government source Existing calibrated and validated model applications at the national or regional
levels Methodology and existing applications documented in peer

- reviewed literature Explicit uncertainty characterization

Run

- time conducive to decision

- support and rulemaking applicat ions U.S. Geological Survey ( USGS ) National
Water

- Quality Assessment Program ( http://water.usgs.gov/nawqa/sparrow/ )

SWAT (Soil and Water Assessment Tool) Environmental Assessment Hydrology, watershed, and water quality model. Used to simulate point and non

point source pollution in user

- defined stream networks (variable scale) using daily or sub

- daily time steps. Federal government source Existing calibrated and validated model applications in large number of watersheds at varying scales

Run

- time conduci ve to decision

- support and rulemaking applications on selected
watersheds Widely used and cit ed in the literature Flexible ways to parameterize effects of a policy U.S. Department of Agriculture ( USDA ) Agricultural Research Service and Texas A&M University http://swat.tamu.edu/

RSEI ( Risk

- Screening
Environmental Indicators ) Environmental Assessment

A nalyze factors (toxic chemical releases, degree of toxicity and size of exposed population) that may result in chronic
human health risks , based on the Toxic Release Inventory

. Federal government source National scope EPA Science Advisory Board

- reviewed Extensive time series (1988

- present) Updated annually Incorporates EPA standard models (e.g., Aermod) and parameters Extensively documented Environmental Protection Agency ( EPA )
Office of Pollution Prevention and Toxics http://www.epa.gov/oppt/rsei/pubs/basic_
information.html

HAWQS (Hydrologic and Water Quality System) Environmental A ssessment Platform to set up and implement SWAT models (see above) using pre

- populated
national data sets Federal government source http://ars.usda.gov/research/projects/proj ects.htm?accn_no=410569

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 44 Table 3

- 2 : Models used to support regulatory analyses.

Data Source Purpose Criteria for Selecting Resource

IPM (Integrated Planning Model) Environmental Assessment and Economic Impacts Estimate impacts of policies on the electricity market, electricity generation, air emissions, etc. Federal government source

Multi

- regional, dynamic, and deterministic linear programming EPA http://www.epa.gov/powersectormodeling
/

BenMAP ( Environmental Benefits Mapping and Analysis Program ) Benefits
Estimate the health impacts and economic benefits of improvements in air quality Federal government source SAB and peer reviewed

Used in n umerous studies published in peer reviewed journ als (e.g., Cohan et al. 2007; Winebrake et al. 2009; Corbett et al. 2007) EPA Office of Air and Radiation http://www.epa.gov/air/benmap/

i

- Tree Vue Benefits

Estimate benefits of changes to urban
forest Federal government source

Peer reviewed

Cited in literature (e.g., Nowak et al. 2006a; Nowak et al. 2006b) http://www.itreetools.org/

IEUBK ( Integrated Exposure Uptake Biokinetic ) Benefits

E stimat e blood lead levels in children Federal government source Use in rulemaking Independent v alidation and v erif ication and a limited peer review of the software
and documentation

Allows for rapid calculations and recalculations of an extremely complex set of equations EPA Superfund Office http://www.epa.gov/superfund/lead/ieubk faq.htm

Leggett model Benefits

Estimate blood lead levels in adults S tate government source (also used by EPA) Use in rulemaking and occupational health standards development Detailed modeling of lead absorption, distribution, metabolism, and excretion for 21 body tissue compartments on a daily time step

A ble to address time

- dependent
conditi ons Leggett (1993) and

California Environmental Protection Agency, Office of Environmental Health Hazard Assessment (Cal OEH HA ; 2013).

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 45 

### 3.1.3 Criteria for Selecting Data Sources
Criteria for selecting data sources for this pro gram include the relevance of the source to the data relative

to the needs , the reliability of the source, and the content. B elow , we discuss considerations we apply in evaluating potential data sources , focusing on the criteria for published data sets

, models , and studies to support benefits transfer

. Note that we assume that the data provided by EPA or its other contractors specifically to support the rulemaking analysis will have been developed under separate QAPPs that reflect the purpose for the analysis ; thus, we will accept the data without further re view. This section therefore applies to data not specifically generated for use in the rulemaking analysis.

Data Sets In selecting data to support rulemaking analyses , Abt Associates prioritize s data from government sources that have been subject to extens ive reviews, are widely used and scrutinized, and are known to be of high quality. We also preferably select national data sets ( e.g., data published by federal government agencies)

, developed using consistent methodology and reporting standards across sta tes , to match the use of the data to support analyses of rules that would apply nationwide. For example, EPA and the USGS provide

water resources data that are national in scope and collected using consistent methods, often with the express intent of makin g the data available to researchers and decision makers

. The data are typically subject to quality review before publication. Databases published by the U.S. Department of Commerce (Census Bureau, Bureau of Economic Analysis), Department of Energy, and oth er federal agencies similarly undergo extensive review before publication.

Thus, most of the data sets identified Table 3

- 1 are national in scope while providing varying level s of geographic specificity (e.g., state, county, metropolitan area, facility, hydrologic unit)

, and are therefore appropriate for use in na tional

- level analyses. Some data are appropriate for general use across analyses (e.g., economic indices or hydrologic network) while others are specific to certain industries affected by EPA actions (e.g., electric power sector, construction sector) , or s pecific locations (e.g., state

- level databases, watershed models)

. See Table 3

- 1 for details.
As appropriate, Abt Associates utilizes data produced by comparable state

- level agencies as best available
when the data are more detailed than federal government sources or whe n the data fit the scope of the analysis (e.g., developing a case study at a selected location)

. For example, state governmental agenci es compile databases of water quality monitoring data for water quality assessment and CWA Section 303(d) listing, and sometimes track implementation of pollution control measures for nonpoint sources

. We used detailed water quality assessment data publish ed by the State of Florida in evaluating the impacts of setting alternate numeric nutrient criteria for Florida waters. Such data were more detailed than data
available from EPA and national

- level sources. For data from all other sources (e.g., peer review ed journal, government

- funded study, trade association study), Abt Associates will select the sources through evaluation of data quality and project needs, in
consultation with the EPA WAM

. T his evaluation will include consideration of EAD’s needs under the Data Quality Act regarding objectivity, utility, and integrity, as described in Section 2.4

. We anticipate

that WA

- specific data needs and possible sources will be covered in a n S QAPP. The EPA WAM and Abt

Associates project team will meet to discuss the identified existing data sources for each project. Through discussion, the EPA WAM will de termine whether sufficient relevant and reliable data sources exist

, and
whether those sources contain sufficient data for the intended use. We will discuss limitations of the data set with the EPA WAM and in method memos, including the timing of th e data relative to the analysis

year , level of resolution in term s of spatial scales or industry categories, important gaps (e.g. , cover only a
subset of conditions to be modeled ), and apparent biases

.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 46 If data gaps remain, we may identify additional existi ng data sources, collect primary data, and/or develop assumptions or a method to address the data gap (see Section 3.1.5 ).

Models

Considerations for identifying potentially relevant models are similar to those for evaluat ing published
literature and databases. The se considerations include assessing whether the purpose , assumptions,

method s , and data requirements of the model fit the information needs of the analysis

, based on review of the documentation, papers on previous model applications in the published literature, and the development status (i.e., whether the model has been peer reviewed). Thus, for models that characterize water quality, we consider the types of water bodies and pollutants simulate d , the data requirements, and reported validity of model estimates in existing applications, to determine whe ther the model may be used to support the environmental assessment and in what application (e.g., national analysis, case study at a specific location). We also consider how the policy may be specified in the model, notably whether the model provides param eters that may be adjusted to mimic the effect of the policy. To analyze the effects of stormwater policies, for example, the selected model needs to be able to mimic the effects of structural

and non

- structural best management practices that may be used t o comply with more stringent stormwater requirements.

To use existing model applications for a different purpose (e.g., using watershed model s previously developed to evaluate the effects of climate change to estimate changes in average conditions resultin g from different stormwater controls in the same watersheds ) , we also review documentation on the model development process, and calibration and validation results

, to evaluate how the specific model application may affect use for evaluating the regulatory scenarios.

Benefits Studies to Support Benefits Transfer Benefit transfer can rely on benefit value or benefit function transfer (BFT) as a source of nonmarket welfare estimates, with benefit functions drawn from individual primary studies in the literatu re. The availability of benefit functions drawn from study sites of sufficient similarity to the policy sites in question is likely to be very limited, particularly for application in national

- level benefit analyses. In cases for which a closely matching study site or policy function is not available, a parameterized meta

- analytic benefit function, estimated using a meta

- regression model of many primary study results, can provide the analyst with a viable alternative. We determine the optimal scope of the metadata based on the exact definition of the dependent variable in the meta

- regression model, which, in turn, defines the set of source studies that can be considered for inclusion in the metadata. The primary tradeof f is often between maintaining close similarity among dependent variables versus including additional information (i.e., observations) in the metadata.
For example, i n compiling resource valuation literature to support meta

- analysis of surface water and open space valuation studies

, we use the following s creening criteria for identifying candidate studies :

 Specific amenity being valued :

 Surface water quality

– The amenity being valued is limited to those in which a water quality change affects ecological services provided by waterbodies, including aquatic life support, recreational activities (such as fishing, boating, and swimming), and nonuse value. For example, a study that estimates WTP for recreational fishing improvements due to fish stocking would not be included in the metadata. However, a study that estimates WTP for recreational fishing improvements due to nutrient reductions would be included in the metadata, presuming it satisfies all other criteria.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 47  Open space (stormwater best management prac tices ( BMPs ) )

– Very few studies directly address values related specifically to stormwater BMPs, some do not estimate dollar values, and most use proxy measures, such as open space, to evaluate effects of these practices

. To develop a valuation function that co uld be used to value an increase in vegetation and open space due to implementation of stormwater management BMPs, we compared features of

stormwater BMPs (including site redesign to reduce impervious cover) to various types of

open spaces considere d in the valuation literature and selected studies of open space similar to open spaces provided by development site redesign to reduce impervious cover ( e.g., increased vegetation; trees and forests; and water

- related, but not “open water”).

 Values estimated

– Following the requirement for welfare consistency, the metadata will not combine studies representing theoretically distinct welfare measures (i.e., stated and revealed data).

 Surface water quality

– W e focus on assembling information fr o m studies using stated

p reference approaches because such studies elicit total household WTP (including use and nonuse values). Revealed preference studies allow the estimation of use values only.

 Open space (stormwater BMPs)

– We limit s elected studies thos e that used hedonic valuation techniques to measure a percent change in home value as most relevant studies for estimating changes in property values.

 Amenity detail

– As described by Johnston et al. (2005), “the study must provide sufficient

information r egarding resource, context, and study attributes to warran t inclusion in the meta data” (p.223).

 Ability to convert environmental quality measures to similar terms

– O riginal studies may use

different metrics to measure environmental quality, t o standardiz e metadata we will convert the
relevant measures of environmental quality change into a single common metric . Some studies

may not be included in the final data set b ecause the environmental quality measure could not be
standardized due to lack of informat ion.

 Surface water quality

- We plan to use the water quality index (WQI) as the metric for water quality improvements within the meta

- analysis, consistent with the existing model. The WQI scale is used to link specific pollutant levels to suitability for human uses. Some studies can be more easily incorporated if they use a standard WQI or water quality ladder (WQL) to measure water quality changes. Other studies may rely on ordinal ranking or verbal description to measure water quality. In these cases, provided available information in the study is sufficient, we will use professional judgment to map these rankings and descriptions to analogous WQI levels

.

 Open space (stormwater BMPs)

- Original studies estimated the value of open space

amenities in ter ms of a marginal change in one of several different metrics: either a home's distance to open space, a home's adjacency to open space, or the proportion of a home's neighborhood kept in open space. To support meta

- analysis we convert all measures into a

si ngle common metric

- percent change in open space within a radius around the property.

 Study location

– All studies must be conducted in the U.S. or Canada.

 Research methods

– Studies must apply research methods that are supported by the literature.

 Duplica tive studies

– Some studies may be released in multiple forms, such as working papers, conference papers, journal articles, and book chapters. Articles will be screened to remove duplicative analysis and analysis that were subsequently revised and publishe d. Multiple values

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 48 can be included for a study, however, if they address a different component of the study data or estimate values for alternative resource changes.

To document meta data w e follow reporting guidelines outlined by Stanley et al. (2013) for meta

regression analyses in economics to record, among other information : (a) the exact databases and other sources searched, (b) the precise combination of keywords, and (c) date completed. In addition, Abt Associates will compile the following study info rmation in electronic format:

 Full study citation

 Study location

 Sample data and description (e.g., size, response rate, income)

 Resource characteristics (e.g., affected waterbody or land type, recreational uses, baseline quality)

 Description of environmen tal quality change, including geographic scale and affected recreational uses (e.g., water quality change from fishable to boatable or percent increase in open space)

 Quantitative measure of environmental quality change

 Original WTP values, that we updat e to current year dollars using the Consumer Price Index,

 WTP estimation characteristics (i.e., parametric versus non

- parametric, inclusion of protest bids and outlier bids, WTP description), if relevant.



### 3.1.4 Data Value Selection Approach
As part of the data source selection and data gap evaluation described above, Abt Associates will work

with the EPA WAM to select data value s (or data sets) from the potential sources identified. We will discuss the selection criteria with the EPA WAM (e.g., age of the data, coverage and resolution, prior use in rulemaking, use in other studies), the relative importance of the criteria in the decision process, and prepare a matrix summarizing how each data source scores according to the metrics. We will only use

metrics that c an be objectively evaluated to ensure transparency in data selection. From the initial review of data content, we may find that:

 One source is the most relevant and reliable

. If we find only one relevant and reliable source,

then we will document the selec tion of this source, subject to EPA WAM approval.

 More than one source offers the data element(s) of interest either individually or when combined

. If we find several relevant and reliable sources we will examine the range of values and recommend an approa ch , subject to the EPA WAM’s approval. For example, we may recommend using the average of values from the multiple acceptable sources or using data from a preferred

source that best fit the data needs (e.g., most recent, most representative ) or other crite rion identified by the EPA WAM (e.g., most cited ; previously used by EPA , for example, for comparative purposes )

. Abt Associates will investigate the range of data values found, whether the data represent normal distributions, or whether the data appear to reflect other distributions. In

cases of multiple data points that are normally distributed, averaging the data for an arithmetic mean is appropriate. In cases of multiple data points that are no t normal ly distribut ed , then using

a medi an of the data may be more appropriate. Finally, when EPA does not require a single value for the analysis, presentation of the value range may be most appropriate. Abt Associates will
work with the EPA WAM in deciding which approach to use for multiple ac ceptable values, and document the process and decision in a memorandum to the file or update to this PQAPP.

 No sources yield the data elements of interest. If none of the so u rces are relevant and reliable, we

may conduct further data searches , obtain data through primary data collection (e.g., phone

contact ; primary survey ), make assumptions or develop a method to estimate the needed value , or

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 49 not estimate the value and describe potential beneficial effects qualitatively

. For example, we developed and imple mented a stated preference survey to estimate total use and nonuse value of aquatic habitat improvements resulting from regulation of cooling water intakes because existing economic studies did not provide a good match to the policy context. We used qualit ative
des cription of secondary benefits such as fishing equipment sales associated with an improvement in recreational opportunities resulting from ELGs and other CWA regulations.

In the case of human health risk estimates, we will identify the endpoints with sufficient weight

- of

evidence for inclusion in estimating benefits. In the case of carcinogens, we may use unit risk estimates provided in the Integrated Risk Information System ( IRIS )

. After consultation with the EPA WAM, we may conduct supplemental modeling (using BMDS) if addition information is needed beyond what is provided in the IRIS documentation.

We may supplement these values, particularly for noncancer endpoints, with concentration

- response functions adapted fro m published epidemiological studies. In this case, for endpoints with sufficient weight

- of

- evidence, we will look for well conducted studies representative of the population under study. We will give preference to studies that treat contaminant exposure as a continuous rather than categorical variable, and adjust for important confounders.

We will document the data selection process in the analysis documentation , including any noted caveat s or limitations that affect use of the data

.



### 3.1.5 Resolving Data Gaps
Up on completing a review and draft selection of data, we discuss with the EPA WAM any quality issues

or limitations that could affect anticipated uses of the data to support analyses. We also discuss possible approaches for resolving data gaps. Methods for f illing in data gaps include, but are not limited to, interpolating between two values, imputing missing values using the average or median of known observations, using data available at an aggregated level of detail to impute values at a finer level of
det ail, and extrapolatin g data from a sample of observations to other observations with similar

characteristics.
For example, i f data are available only at a more aggregated level of detail than EPA is seeking for the analysis (e.g., Hydrologic Unit Code ( HUC ) 10 versus HUC12 level ; 4

- digit North American Industry Classification System ( NAICS ) compared to 6

- digit NAICS level), we would discuss w hether to do the analysis at the more aggregated level, or disaggregate data to the finer level of detail using proxy metrics, recognizing the uncertainty involved in doing so. In modeling the effects of stormwater controls on water quality, for example, we used projected development data at the level of HUC12 basins as input to a watershed model that relies on developmen t at a finer level of detail . Mapping the development to modeled subbasins required distributing the projected development in proportion to the baseline distribution of development across the subbasins.

In addition, as described in Section 3.1.4 , it may not be feasible to resolve data gaps. In such cases, we fully discuss the data limitation in a section of the report entitled Limitat ions and Uncertainties, including the likely resulting direction of bias, and any information on the potential magnitude of the bias, on the final cost, benefit, and economic impact estimates.



### 3.1.6 Data Gathering Documentation and Records
Data gathering documen tation and records depend in part on the type of existing data.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 50 Published Literature Abt Associates collects and organizes literature using reference management software (EndNote) when
the literature review task involves more than a few articles

, or cover s topics that are germane to several project activities and thus may become a data source for use in later work assignment s

. In particular, we use reference management software to maintain an up

- to

- date repository of the literature on topics germane to the types of analyses or methods often conducted for this pro gram , such as valuation methods for different benefit categories, dose

- response functions, water quality indices, and statistical procedures

. Using reference software allows for efficient compilatio n of full bibliographical information, facilitates citing of sources in final reports, ensures consistency in the way metadata information is recorded, and minimizes opportunities for transcription errors when citing literature.

Abt Associates’ Research A ssistants, Associate Analysts, and Analysts are typically responsible for identifying and screening the relevant literature

, and for summarizing the data or findings relevant to the

research task. The team accomplishes this by noting relevant aspects of th e study specific to each research question. For example, the research notes may include the period of the study, type of waterbody, spatial or temporal resolution or scope of the study, pollutants evaluated, study method, conclusions, and important limitat ions or uncertainties that may affect the use of the study results in other analyses (see

Section 3.1.2 )

. As described in Section 3.1.3 , f or meta

- analyses, we follow reporting guideline s outlined by Stanley et al. (2013) for meta

- regression analyses in economics to record, among other information : (a) the exact databases and other sources searched, (b) the precise combination of keywords, and (c) date completed.

The Abt Associates WAM, o r a designee

, is responsible for verifying the accuracy of the data extracted from the original source. We may accomplish this verification by independently reviewing a selected number of papers and comparing the literature review notes

, or by interviewing the reviewer to gauge understanding of the fine points of the study and applicability to the target analysis for which the literature review is being conducted.

We often transcribe v alues from published articles through electronic cut and paste, or manual ly . As a result, transcription errors may occur . We verify the accuracy of values extracted from the literature by directing a second individual, designated by the Abt Associates WAM, to confirm that (1) the value used in a spreadsheet or cited in the docu ment corresponds to the value in the original source, (2) the units (and unit conversions) are correct, and (3) any confidence interval or uncertainty bound provided in the original source

, are included. We perform this transcription QA as part of the spreadsheet or document QA review discussed below.

Electronic Data Sets Abt Associates collects and organizes electronic data sets in project folders organized by task or analysis components. We retain copies of all data in the original “as received” format, along with any associated metadata, related publications, source Web page, email message, and other source information. Table 3

- 1 provides the criteria for selecting the electronic data sets commonly used under this contract. Section 4.1.4 describes our procedure s for databases and data set transfer and maintenance.

Telephone and Email Data Collection

Abt Associates documents conversations used as “data source s, ” and generates corresponding electronic files in the project folder. Abt Associates maintains the follo wing documentation procedures when collecting data that may be used to support analyses conducted under this project via telephone and email:

 Document telephone contacts using a Telephone Contact Report depicted in Appendix D

.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 51  Save email contacts as portable document format ( PDF ) files

, and add to the electronic project

file.

The Abt Associates WAM and EPA WAM will determine which data materials gathered through telephone or email conversation during the project need to be included in any record.

Hardcopy Documents

In the event that we receive data a s hardcopy documents , Abt Associates scans the relevant pa rts of the
document and works with the data in the same way as described above for published literature. The project team retains a hardcopy of the document in physical folders and provides the document to the

EPA WAM at the end of the work assignment

.



### 3.2 Dat a Management

This section describes the procedures and best practices that Abt Associates employs to manage data

under the individual project s

.



### 3.2.1 Standardization of Data Elements
In developing analytic frameworks, Abt Associates defines standard categories that will be used throughout the analysis to ensure consistency in concepts and alignment of data with respect to temporal, spatial, industry sector , and other dimensions. Establishing and using standard categories is a best practice

that ensures uniform r eporting of data across the various parts of the analysis.
For each needed adjustment, we document the adjustment factor or crosswalk used to convert the secondary data into the framework. For transparency and reproducibility, we retain in our files the

or iginal value provided in the data source , along with the adjusted values.

Table 3

- 3 summarizes standard data elements for analyses conducted for EAD.

Table 3

- 3 : Standard data definitions for EAD analyses

. Standardization category Standardization

basis Implications for handling of secondary data

Temporal Analysis year If needed, we bring data forward to the analysis year. For costs, we use
C CI, ECI or other appropriate indices or adjustment factors, depending on the type of cost (see Table 3

- 1 )

. Spatial State or counties We code data using the state postal abbreviations or county FIPS as
identifiers

C om ID We code waterbodies using their C om ID ( if doing the analysis at the

NHD level )

, reach ID ( if using E2RF1) , or other unique identifier (e.g.,
Florida DE P WBIDs).

HUC We code watersheds using the HUC system. We identify the appropriate

HUC(s) in ArcGIS as needed.

NERC region We code electric power grid region s using EIA’s NERC regions

definitions.

Economic Sector NAICS If needed, we categorize data provided for facilities or economic activities using the appropriate NAICS sector (s )

. Chemical CAS RN For analyses involving calculations across chemicals, we use the CAS
RN to identify unique chemical s

. Electric power generator EIA ID We identify the EIA ID for electric power generators , operators, etc. to
facilitate cross

- references to DOE data on the operational characteristics of the plants.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 52 Table 3

- 3 : Standard data definitions for EAD analyses

. Standardization category Standardization

basis Implications for handling of secondary data

CAS RN = Chemical Abstract S ervice registry number

CCI = Construction Cost Index

C om ID = Integer value that uniquely identifies the occurrence of each feature in the NHD DEP = Department of Environmental Protection DOE = Department of Energy EIA = Energy Information Administration

EC I = Employment Cost Index HUC = Hydrologic Unit Code NERC = North American Electric Reliability Corporation NHD = National Hydrography Dataset WBID = Waterbody Identification

During the development of the framework, we also define the units we will use to conduct the calculations, summarize the data, and present the results (e.g., land area used as hectares in the watershed
model but presented in acres; loadings calculated in kilograms but presented in tons). As needed, we convert data from their existing units to the calculation or reporting units using universal conversion factors (e.g., obtained from Excel or Google built

- in unit conversion tools, or onlinecon version.com ).



### 3.2.2 Data Entry

Abt Associates Research Assistants, Associate Analysts, and Analysts typically are responsible for collecting existing data and developing the associated databases and spreadsheets. Prior to us ing the data in any analysis, the Abt Associates WAM or a designee review s the data for accuracy and appropriateness. For data entry, we check and document that the data and other information accurately represent the original source by verifying 100% of manually entered data points

.



### 3.2.3 Merging or Uploading Electronic Data from Existing Sources
The steps for ensur ing the quality of the data extracted, merged, or compiled from electronic sources depend on how the data are to be incorporated in the analytic frameworks.
If practicable, we input data directly into the analytic framework from source files, with the analysis framework or model providing a unique link to any shared information

– i.e., the analysis relies on the same data stored in a unique location to avoid potential inconsistencies.

Af ter extracting or merging data from existing sources, we verify the following :

 The extraction program did not generate any errors (e.g., if importing into an Access database, the software program did not warn of values that could not be converted into any new data format) ;
 The number of records extracted match those in the original source (i.e., the program used to

extract or store the data did not truncate the data set);

 The row and/or column totals match those in the original source or in a summary report that describes the original source;

 All columns have valid values , as verified by visual inspection (e.g., numeric values rather than blanks, or no negative numbers, based on knowledge of the file contents) ; and

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 53  Data in selected records match those in the original source exactly , as verified by manually comparing selected records (5

- 10 records in the case of the process is automated and record formats are consistent throughout the data set)

. These procedures enable us to confirm that the extraction pro gram did not generate any errors. We describe additional data manipulations to ensure that the extracted data use standard categories

, and are consistent with the analytic framework in Sections 3.2.1 and 3.3.2 , respectively.
In addition to the merged data, we retain a copy of all existing data used in analyses as received from the original source on a particular date , documenting all changes to the original values, along with the reason for any adjustments (e.g., unit conversion, standardization to analysis year). Abt Associates staff cite the date and origin of the data in all spreadsheets, databases, and documents, and provide hyperlinks to online

sources for e asy reference. We use this meta data during our quality assurance review of the framework to
assess data sources and

, at later stages of the project (particularly when analyses may take place over several years), to identify input data sets that may need to be updated based on the publication of more
recent data.



### 3.2.4 Data Review

Sections 3.2.2 and 3.2.3 describe Abt Associates’ procedures for verifying that the data values we enter manually, or download electronically, match the original source. After data verification, we review the
data to identify any remaining data gaps, zero or missing values, values with QA issues, or outliers (e.g., values outside the expected range, negative values, or values orders of magnitude higher than reasonable).
We correct such values whe n the data exist to do so, or remove them from the analysis, documenting in the file the reason for the change or exclusion.



### 3.2.5 Data Storage and Manipulation
Abt Associates compiles existing data in spreadsheets (e.g., Microsoft Excel), relational databases (e.g., Microsoft Access, MySQL), reference management tools (e.g., EndNotes), and other software tools (e.g., ArcGIS, R) to facilitate their use in analyses

. Abt Associates has dedicated information technology (IT) staff responsible for the installation and maintenance of all hardware and software infrastructure

, includ ing automated daily incremental backup of all data stored on networked resources. The Abt

A ssociates WAM will work with the IT staff to ensure that data storage procedures are supported by the

IT infrastructure. At the completion of this contract

, Abt Associates will archive the data (including analysis results) so as to preserve the se data with no further modifications.

Abt Associates stores and handles e lectronic files containing CBI data i n accordance with our CBI Security P lan for this contract , which ensures that only staff with need

- to

- know have access to the files and any derived analyses that are also CBI

. Abt Associates employs best practices in the use of softwa re to store and manipulate data, such as using queries and lookups in Access and Excel to extract relevant portions of datasets, and including

intermediate results to facilitate review

. The PQA Advisor for an individual work assignment, or his or her designee, reviews all calculations within spreadsheets and databases by hand to verify that formulas are correct and producing the intended result

, and hence the data in the columns are correct

. For example, s preadsheet best practices minimize the risk of errors through structured design and consistency; such practices also ensure transparency and efficiency in update time and costs. As an

e xample, the Spreadsheet Standards Review Boa rd ( http://www.ssrb.org/ ) develops and publishes spreadsheet best practices that cover 16 areas of spreadsheet model development . These include general concepts; workbook structure; sheet structure; formats and styles; assumption entry interfaces; sensitivity analysis; outputs and presentations; calculation formulae; naming principles; time series analysis; checks;

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 54 printing and viewing; multiple workbooks; security and protection; Visual Basic programming; and

miscellane ous. These guidelines are available to all staff for implementation as applicable, although not all areas may be applicable to any one project. Therefore, Abt Associates staff tailor implementation to the complexity of the existing data and forthcoming ana lysis, while maintaining the framework for revising and adding data and complexity at minimal cost.



### 3.3 Data Quality Assessment and Use

After identif

ying and selectin g existing data sources, Abt Associates and the EPA WAM review the data to assess usability f

or the analysis

. As described in Section 3.3.1

, we compare our findings regarding data accuracy, reliability (e.g., presence of apparent outliers or unresolved questionable values), and representativeness (e.g., completeness, resolution

, variability) , against the objectives

. We document the limitations and gaps identified during our internal review in a memorandum to the EPA WAM, along with proposed approaches for addressing these limitations, whe n possible. We then decide with the EPA

WAM on approaches to be use d to address the limitations

. These approaches may include moving ahead with the analysis using the existing data and detailing the limitations , along with implications for interpreting the results, in the final deliverable describi ng the analysis ; collecting additional data targeted at specific gaps (e.g., collecting additional data for a selected region for which the national dataset is lacking or has questionable values) ; or pursuing primary data collection.



### 3.3.1 Data Quality Verific ation and Data Quality Reporting

Abt Associates staff will evaluate existing data against objectives of: (1) accuracy, (2) reliability, and (3) representativeness, considering the data source, available documentation, and content.

Abt Associates assume s t hat data and informat ion contained in s tate and federal databases and reports are accurate and reliable to the extent specified in the quality assurance and control documentation accompanying the source. For these sources, we assess and identify any known limitations or caveats

, and

summariz e the data to confirm the scope and identify possible outliers (e.g., data that are not consistent with the body of information collected) and other data issues that may affect use of the data

. For other data sources (e .g., peer reviewed literature, trade publications)

, Abt Associates reviews the

documentation and content to identify potential f or bias es or other data quality issues (e.g., data are based

on a limited sample)

. W e consider the quality of the documentation, clarity of presentation, discussion of uncertainty, and scientific merit of the method s and assumptions to assess whether the data source is of sufficient quality for u se in the analysis, and if so, note any limitations. Abt Associates will communicate

th ese data quality issues

, and discuss the best approach to address the issues, including gathering data from several such sources and varying conditions.
Abt Associates will report the outcome of data quality assessment to the EPA WAM in meetings or short written deliverables (e.g., memoranda, tables), as needed, and record findings in the project file.



### 3.3.2 Use/Analysis of Existing Data /Data Operations

Abt Associates uses e xisting data to describe baseline conditions

, and as input to environmental and

economic models for evaluat ing the impacts of regulatory options. This section describes how we use the
data within these frameworks , focusing specifically on data for environmental and economic models

. We may implement these models in a variety of software, such as spreadsheets, structured programs, and relational databases).
We follow specific procedures when conducting complex modeling assignments, such as when running
watershed or economic impact mod els to support rulemaking

. ( These procedures embed documentation
elements also described later in Section 3.3.3 .)

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 55 At the onset of each scenario dev elopment, we prepare a table detailing the various elements of the scenario specifications: watershed(s) or plant universe , model run years, output parameters, version of the model ( e.g., date of executable file), analysis baseline ( e.g., date input was re ceived), list of input files

modified since the last model run , changes to default model assumptions, etc. We use this table to record metadata for each scenario, and to track progress in completing the model runs

. W e implement the steps in the modeling pr ocess using algorithms

, batch programs (in SAS, R, VisualBasic, or other scripting language) , and spreadsheets, to prepar e model input data , run the model, and post

- process the outputs automatically , or at a minimum perform several interim steps automatica lly

. We generate log files as part of the batch program to audit each step of the process and verify that the scenario was specified correctly when reviewing the results

. We also build in validation checks at various stages in the algorithm or the spreadsheet to flag any conditions that could affect the analysis. For example, we build in QA by including code in the algorithm to verify that interim or final calculations fall within a valid range (e.g., BMPs have calculated drawdown times of at least 48 hours, all values are positive) and to generate an error when the conditions are violated.
When developing or modifying an algorithm or batch program, we verify that the program perform s the calculations correctly by reproducing the calculations offline (e.g., in Access or Excel) on a sample data

set and/or verifying known equalities or values in internal validation calculations within the program. The approach d epends on the type of algorithm being developed or the extent of the modifications

. We review algorithms and calculations by having a second person not directly involved in developing the algorithm/program review the code to ensure its accuracy. T he Abt Associates WAM is responsible for determining the extent and methodology for the review. T hus, w e may review major algorithm development and modifications by doing the calculations offline. For relatively minor changes, such as changing assumptions or inserting conditional statements within a largely unchanged algorithm (e.g., adding a condit ional clause to subdivide treatment of conditions using If

- Then statements), the Abt

Associates WAM may decide to approve the changes after conducting a code review

, without requiring reproduction of the calculations offline.

We conduct f

irst

- level qualit y control of model runs by verifying that all steps involved in preparation of the input files and processing of the model outputs were completed correctly. In conducting this review,

we consult the log file generated by the batch program s and any additional log files generated by the

model

. We process the raw model outputs using an algorithm and batch program customized to the particular model outputs (e.g., SWAT, cost impact model). For example, we process model outputs to generate the desired statistics (e.g., average daily water quality index score for each analyzed reach, number of entities exceeding cost

- to

- revenue thresholds) along with intermediate calculations and summary statistics to allow auditing of the results. Thus, analysis of daily reach

- level outputs that we obtained from the SWAT

model includes min imum , max imum , average, median, and quartile statistics for each water quality parameter and flow to allow comparison of the distribution of daily values and not only indicators of

central tendencies

. We review these outputs to assess whether the results are within the expected range. For models that
generate a large number of observations (daily values on all reaches analyzed over a 30

- year period), we

look at indications that t he distribution of values is reasonable by looking at the minimum and maximum values. Once the data have passed this initial check, we look at the changes across scenarios (or relative to different runs of the same scenario) to assess whether they are in t he anticipated direction.

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 56 We save summary results, raw model outputs

, log files, any model input files modified in running the particular scenario , and the algorithm used in analyzing the scenario in a separate folder

. Keeping all files together ensures t hat we can reproduce the results at a later date if needed.



### 3.3.3 Methodology Documentation and Conceptual Review
In developing methods for data analyses, Abt Associates documents proposed approaches and options in
methods memorand ums . The Abt Associates WAM m ay involve the Project Manager in developing the methods, and involves the PQA Advisor in review of the memorand ums . The Abt Associates WAM provides the memorandums to the EPA WAM, who reviews and comments, and involves the EAD
Branch Chief in selecting th e approach. Abt Associates typically prepares method memoranda in a format that facilitates insertion into the final report or technical support document for the regulatory effort.

The PQA Advisor or his or her designee reviews the spreadsheets, models, and programs against the methods memoranda when performing and documenting quality control (see Appendix A for the checklist template we will use f or documenting PQA Advisor reviews under this contract )

. The WAM adjust s the

checklists as needed based on the complexity of the analysis to account for the flow of data from the
source data t o final deliverables, including location of th e spreadsheets, da tabases, and computer

program s; staff w ho ran th e analyses ; location of the input data ; version of the data ; and w hich iteration of analytic program or frameworks resulted in the numbers that we report.
We also use the following procedures, as appropriate to the analysis, to document the methods and flow
of the work :
 Document the process (flow) of the data work in the appropriate task

- level directory, including :

 Name and location of input files

 Name and location of program

 Name and lo cation of output files or tables

 Date the program or process was run

 Document the order of steps within spreadsheets, programs, or separate file within the appropriate

task

- level directory

.  C atalog incoming and outgoing (sent outside the project) data files , including name, source, and

date

 Maintain crosswalk s between report table s and origin (e.g. Excel workbook)

 Include documentation within analytical framework or program code, including:

 G eneral information at the top of every framework or program, such as name and path of the program, the name of the project, the name of the programmer, names of input and output files, and the purpose of the program

 T itles and/or footnotes in any output

 S ufficient comments to provide a ‘roadmap’ to the analysis flow

 T rack processing steps a nd program runs

 S tore all output s of analysis runs

 A rchive programs and all project documentation.
The Abt Associates WAM, working with the project

- level PQA Advisor

, also reviews that the analyses implement the methods described in the method memoranda

, i ncluding any test or pilot runs to verify that the methods effectively produce the analytical result agreed to with the EPA WAM. The Abt Associates

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 57 project team discuss es any issues regarding the conceptual review with EPA WAM, and documents any agreed

- upo n changes to the methods in memoranda, as needed, and monthly prog r ess reports.



### 3.3.4 Technical Review of the Data Analysis This section describes Abt Associates’ process for ensuring that project staff are properly performing data analysis to support the proje ct.

Section 3.2.5 describes the electronic and manual spreadsheet best practices and quality control checks that Abt Associates staff will use to verify the accuracy and usability of data to meet the project

o bjectives. Abt Associates k eep s record of spreadsheet reviews, including the name of the review and the date of review completion, within the spreadsheet coversheet. Abt Associates analysts will also perform analyses of existing data within spreadsheet wor kbooks consistent with the best practices described above. As such, under best practices, existing data will be separated from assumptions, outputs, and presentations, which facilitat es data verification. Appendix A show s the checks that the PQA Advisor or designee, will be responsible for completing related to assumptions, summary statistics, models/algorithms; formulas; outputs; and presentations t hat we compile and develop within spreadsheets.

The WAM will add WA

- specific detail to the QA template, as needed, to facili ta te PQA Advisor review.

A ny time that we make material changes to the models or calculations, the Abt Associates WAM or designee reviews the inputs, calculations (including correct reading/writing of the inputs and outputs), and outputs to ensure that we have processed the data correctly. For models f or which calculations have already been verified, the review focuses on the model inputs to ensure that they are accurate, and on the outputs to ensure that they are complete, consistent, and properly described.



### 3.3.5 Final Verification of Data Analysis and Reconciliation with User Requirements

T his PQAPP identif

ies the level of review required for project deliverables

, and who will conduct the
review. Abt Associates conducts t his review us i ng the quality assessment tools described in Section 3 .3.1 of this PQAPP.
Upon discovering issues that affect any delivered work product, the Abt Associates WAM immediately

informs the EPA WAM of the change and of a ny resulting modifications to QA procedures to prevent recurrence of the problem. If needed, Abt Associates revises the P QAPP to reflect changes in the procedures.
The Abt Associates WAM also immediately informs the Quality Assurance Manager , as well as th e PQA
Advisor and Project Manager

, of material errors that passed the established QA process

, to identify and evaluate mitigating measures to prevent recurrence and resolve the issue

. Abt Associates includes a summary of QA

- related activities as part of its monthly progress reports , in a
section titled: “ QA Activities during Reporting Month .” The summary describe s steps taken to assess the quality of data used in the analysis, verify that the data, analysis frameworks

, and models used to support

programma tic activities meet the data quality objectives and are appropriate, and verify the accuracy of
deliverables to EPA

. For example, we will indicate when files have been reviewed

, and document any changes to the analysis or draft/preliminary results. Prepari ng the monthly progress report s prompts the
Abt Associates WAM s to identify and report on the QA activities conducted during the month , and consider the QA activities that should be performed in the next reporting period.
For example, QA activities report ed under WA 05, Steam Electric ELG, include:

 Compared most recent Energy Information Administration ( EIA ) data (for 2012) to data used in the existing analytic framework (EIA 2009 and Steam Electric Survey) to identify possible issues

### Existing Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 58 before incorporating the new data (e.g., change in ownership, retirements). Abt Associates will describe and review these issues with the EPA WAM before bringing new data into the
framework.
QA activities reported under WA 03, Post Construction Economic Analysis , include:

 Perf ormed additional QA/QC of all models and input /output data by using cross

- walks across different parts of the analysis to ensure consistency

, and by using spot check/manual calculations
to validate model results (e.g., compare maximum BMP storage volume p rovided in model output file to value calculated manually using input specifications for BMP capacity and development area managed (storage depth ×total development area ×fraction of development managed by the BMP), check mass balance on the BMP by compari ng total water inflows to the total water outflows and changes in water storage )

.  Verified that exploratory analyses of SWAT model outputs were done correctly (e.g., data are extracted from the correct file(s), data represent the desired metric, necessary unit conversions are implemented correctly, and summary statistics are properly calculated), the analysis approach provides insight relevant to each study question, and additional questions emerging from the analysis are identified and either resolved or f lagged for EPA attention.

 Verified that extrapolation approaches are implemented correctly (correct data extraction, appropriate model specifications), and that the results are appropriately described and qualified.

 Performed QA/QC of small

- scale model inp uts to ensure that land use change and policy scenarios are implemented correctly and are consistent with inputs for larger scale model (e.g., land use changes reflect projected development, differences between the amount of development
in the small vs. la rge

- scale models are consistent with the relative watershed vs. subbasin areas).

 Pulled data for selected watersheds and model components, and compared results to engineering outputs and across locations and scenarios to understand conditions and processes that contribute to increased loads and/or concentrations simulated in SWAT. Corrected equation used to reset phosphorus limits in SWAT when effluent concentrations are specified for a given BMP; under certain circumstances, the calculations could result i n division by zero. Reran SWAT models. Change did not materially affect results obtained previously.
The Project Manager for this contract holds bi

- weekly meetings with all Abt Associates WAMs. These meetings provide an opportunity to exchange ideas on bes t practices, QA issues, etc. that affect programmatic activities under this contract. Additionally, the Project Manager and senior managers meet once per quarter with the Division Vice President to discuss contract status, including any QA issues that

may affect performance under the contract , and resolve any issues by taking appropriate actions at the company level

. Abt Associates includes a QA section in final project deliverables (e.g., major rulemaking support documents, final project reports) that addresses the data collection, production, analyses, and use

.

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 59 

### 4. Primary Data

This section describes Abt Associates’ procedures for data gathering, management, and quality assurance and use pertaining to primary data ; primary data are new data generated specifically fo r use in the

pro gram

. Abt Associates will support EPA in developing, administering, and analyzing results of electronic questionnaires to gather data from facilities in specified point source or industry categories to support
E LG s

. This support corresponds to steps 4 and 7 in the ELG development flowchart of Figure 2

- 2

. The EPA WAM will define the industry, source category, or other entity or population subject to a data collection effort.

Figure 4

- 1 illustrates the general process for developing and implementing industry surveys. We discuss considerations for data gathering and d ata q uality a ssessment and u se that are pertinent to the conduct of industry surveys in Sections 4.1 and 4.2

. Figure 4

- 1 : General Process for Developing and Implementing Industry Survey.

In addition, EPA may also direct Abt Associates to develop and administer smaller survey efforts to collect specific primary data from a subset of entities 2 or may direct Abt Associates to collect primary
data from the public to support monetization of benefits (e.g., stated preference surveys)

. The sections below focus on industry surveys and small data collection efforts. We describe the general pr ocess for conducting interviews to collect primary data in Section 4.3

. We will address additional procedures for conducting stated

- preference surve ys, if requested by EPA, in a work

- assignment specific SQAPP.

2 A survey of nine or fewer entities for the same information does not require an Information Collection Request (ICR)

. Primary Data

 Industry s urveys

 Industry c ontacts and

s ite v isits

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 60 Note that the statement of work under this contract does not involve laboratory or field measurement; as such

, this PQAPP does not address elements specific to laboratory analytical methods, eq uipment, and supplies.



### 4.1 Data Gathering
This section describes data gathering elements for primary data collection that differ from those for existing data collection.



### 4.1.1 Sampling Process (Experimental) Design

T he experimental design and process for collect ing primary data will depend on the following factors:

 Type s of entities and data collection instrument(s)

. For industry surveys, EPA and Abt Associates may perform data collection using one or more survey instruments within the same or consecutive phases. EP A may choose to distribute screener and/or detailed questionnaires. EPA uses screener questionnaires to identify entities that may be subject to a regulation

, and/or to identify a subset of entities that should participate in detailed surveys to help reduc e the burden of a full survey
effort. As an example, for the CWA section 316(b) rulemaking, EPA conducted a screener survey to identify electric power plants with cooling water intake structures that would likely be s ubject to a 316(b) regulation. EPA subs equently targeted these facilities in its more detailed questionnaires to support the rulemaking effort. For the Steam Electric industry survey, EPA did not use a screener survey but distributed the questionnaire to all coal

- and petroleum coke

- fired

power plants, and to a statistically sampled subset of the rest of the steam electric industry, including oil

- fired, gas

- fired, and nuclear power plants in a single phase.

 Types of d

ata . EPA generally develops multiple questionnaire parts and sections as need ed so that questions on a similar topic can be grouped together. EPA may also direct questionnaires at specific types of entities (e.g., publicly owned treatment works recipients to assess the impacts of pretreatment). As an example, the Steam Electric que stionnaire consisted of nine parts (Parts A through I) with different parts targeting certain types of power plants or power plant operations.
Abt Associates assisted EPA with developing a specific section covering f inancial and economic

information applic able to all types of plants

.  Data collection instrument f orm at(s) . EPA has used both paper (hardcopy) and electronic questionnaires in industry surveys used for past ELG development efforts. Because of the efficiency of electronic questionnaire s , we antic ipate that EPA may conduct future industry

surveys under Contract EP

- C

- 13

- 039 using electronic questionnaires which will either be mailed to respondents or delivered over the Internet

, if possible given data security requirements (e.g.,

ensure CBI protection)

. Accordingly, we focus our discussion of data gathering elements in this PQAPP on electronic questionnaire s . In addition, for small

, directed data collection efforts , such as to collect pricing or other information from vendors , EPA may cho o se to conduct telephone surveys. In these cases, Abt Associates will gather responses onto a data collection form developed specifically for the purpose

. We describe our general procedures for conducting primary data gathering interviews in Section 4.3

. 3

3 In the un likely event that larger collection efforts need to be conducted by telephone, we will discuss the applicable QA procedures in a project

- specific SQAPP. For example, Abt Associates would program the data collection form into Computer Assisted Telephone Int erviewing system (CATI). CATI minimizes interviewing

error , through its automatic skip patterns and range checks, which has the added benefit of minimizing post field editing. Specific d ata collection protocols for phone interviews will be determined based on a discussion with the EPA work assignment manager and will be addressed in the SQAPP.

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 61  Procedures for h andling CBI r esponses

. We will handle a ny technical and/or financial information designated CBI in a questionnaire response or gathered during phone conversation according to Abt Associates’ CBI security plan

. The potential for the primary data to be classified

as CBI will affect the questionnaire format

, methods for transmitting data to EPA and Abt Associates, and subsequent p rocess ing of the data in the analyses.

Abt Associates will work with the EPA WAM, the EPA statistics team, and any other relevant contractors (e.g., for engineering analysis) to develop the primary data collection process. Generally, the EPA

statistics tea m develops the survey sample and weights

, and provides these data to Abt Associates for use in implementing the survey. Abt Associates may assist EPA in defining the sampling frame

– the list of all entities that can be sampled

– based on the goal of the c ollection efforts by providing information on the number and characteristics of regulated entities. For example, the sampling frame may be stratified by industry sector, geographic area, or other characteristics (fuel type, economic activity performed, num ber of employees, etc.). Abt Associates will rely on publicly available data (i.e., based on existing data described in Section 3) to support EPA in identifying the appropriate sampling frame for industry surveys. In past primary data collection efforts, Abt Associates also helped EPA develop the data collection instrument (questionnaire), supported development of the Information Collection Request (ICR) as needed, provided helpline support to data providers, and reviewed and processed the primary

data. Fo r smaller data collection efforts involving fewer than ten respondents, Abt Associates identifies the entities based on the purpose of the request, collects the information, and summarizes the results for use in the analyses.

We will document the data col lection process in a series of planning memoranda and/or documents

, including input for developing any requir e d ICR Supporting Statement.



### 4.1.2 Data Gathering Methods and Steps
Data gathering methods and steps will depend on the type of data to be gathered (facility information, pricing data ) and Abt Associates’ role in the effort

. 4 Below, w e describe general QA

- related considerations and procedures for industry surveys and other primary data collection efforts carried out by
Abt Associates to support ELG d evelopment

. We will describe additional procedures that are specific to a given project or to other types of primary data collection efforts, such as stated

- preference surveys

– in a SQAPP

.  Identify information to be collected and purpose : During the design phase, Abt Associates develops a list of data to be collected from the respondents, based on the analytic blueprint and

data gaps identified during review of available secondary data. We also outline alternative to using primary data, if any. We then identify the desired data resolution (facility/immediate

owner /parent owner , time period, budget component , etc. ), units, additional considerations that

c ould affect the way the data will be collected (e.g., A re the respondents likely to have the requested information at the desired level of detail? Are categories or ranges sufficient or do we
need actual values? )

. We describe the results of this assessment in a memorandum to the EPA
WAM for review and approval prior to proceeding with develop ing the survey instrument.

 D esign instrument to minimize burden and potential for errors : Abt Associates designs questionnaires to collect the necessary information while minimizing respondent burden and

4 For some industry surveys, Abt Associates may be conducting only some of the activities enumerated. For example, for the Steam Electric industry survey, a differen t contractor was responsible for mailing and receiving the surveys. Abt Associates supported preparation of the questionnaire and sample frame, and received and processed the economic and financial sections of the completed survey.

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 62 potential for error

. We review the literature and secondary data collection instruments to identify industry

- specific terms that would be familiar to the respondents. W e develop electronic forms that incorporate validation rules to minimize the potential for invalid entries an d ensure data consistency. For example, we may use Excel forms to request information from industry respondents and use the built

- in data validation feature to limit entries to a pre

- populated list (e.g., NAICS codes, state abbreviations, pre

- determined ra nges).

 QA and test instrument : W e pretest the questionnaire using staff not involved in the development of the questionnaire and

, if feasible and approved by EPA, using respondents selected to approximate the target population (e.g., trade association or i ndustry representatives) . Pre

- testing the survey instrument validates the adequacy, clarity, and balance of the data collection instruments, both for interviewers administering the questionnaire (if relevant) and the respondent answering the questions.

 Ob tain the respondent list

. As noted above, Abt Associates will typically receive the survey sample from the EPA statistics team. If needed, Abt Associates’ sampling statisticians can provide support in developing a sample design that meets the project’s ana lytic objectives and

resources. If needed, we will detail this activity in a project

- specific SQAPP.

 Develop survey tracking database : Once EPA has determined the final sample, Abt Associates develops a tracking database of respondents in Excel, Access or other similar software . We use this database throughout the survey process to track status and actions related to each response (e.g., survey sent, response received, response reviewed, follow

- up, QA completed, etc.)

.  Mailing : For mai l survey s targeting large samples , Abt Associates uses a three

- step process to maximize the response rate (as specified in Dillman et al. , 2009) : (1) send a pre

- alert letter announcing the forthcoming survey, explaining its purpose and process ; (2) mail th e survey package, which includes a cover letter, questionnaire, and business reply return envelope; and (3) follow up to increase the response rate. To maximize response rates, it is essential to allot at least 8 weeks for survey administration; avoid sche duling mailings during a holiday season (especially between Thanksgiving and Christmas); and limit the length of the survey.

 Follow up with respondents to increase timeliness and response rate : Industry survey

r espondents are typically compelled to return completed questionnaires within the deadlines specified by EPA. We track responses received against the list of respondents that were sent a survey. As the deadline approaches, we contact respondents to ensure that they are able to complete the survey on t ime. For other surveys where participation may be voluntary, our typical approach includes making at least three additional re

- contact attempts after an initial contact

failure.

 Setup and maintain telephone and email helpline to answer respondent questio ns : Throughout the survey implementation period, Abt Associates set up and maintain a dedicated toll

- free telephone
number and email account to answer questions on the survey. We enter e ach call and email into a tracking sheet and record the date/time, sur vey respondent ID, contact information, detail s of the question, name of Abt Associates employee answering, and resolution.



### 4.1.3 Completed Questionnaire Handling and Custody
Abt Associates may receive completed questionnaires directly from the respondents, from EPA, or from another contractor (e.g., engineering contractor). Upon receiving a completed questionnaire on CD

- ROM
or other electronic media , we log the receipt in our tracking database , verify that the file is readable, and acknowledge receipt to the sender

.

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 63 We maintain material in their “as received” state (e.g., original media)

, and create electronic copies of the files in the project folder for further review and analysis (see Section 3.1.6 ) . We store the material in accordance with any applicable CBI procedures as detailed in the CBI Plan for Contract EP

- C

- 13

- 039

.



### 4.1.4 Data Management
We describe general data management procedures below; s pecifi c data management procedures that vary depending on the type of primary data collected may be covered in a work assignment specific SQAPP

. Abt Associates typically receives industry survey responses as separate files for each respondent (e.g.,
individual Excel workbook for each plant)

. Abt Associates consolidates the se data into a Draft Database

to facilitate review and use in eventual analyses. Depending on the number of responses and complexity,

we manage the consolidated data using spr eadsheet or relational database software (Excel, Access) . For example, for the Steam Electric industry survey, Abt Associates compiled the information from over 700
individual files into a Microsoft Access database

. We used the database to organize the sur vey data into a series of indexed tables

, and enforced relationship and integrity checks to ensure data consistency at the plant, owner, and unit level. Indexes and relational integrity checks ensure s that the data are internally consistent (e.g., informat ion is exactly the same across tables, where applicable)

, and help prevent duplicate records, for example by ensuring that only one record exist s for each unique respondent or combination of respondent and other parameter (e.g., plant

- year).

We include a d ata dictionary and metadata in the database to clearly define the content of each table and

field , and document the source of the data (file name, dates created/imported/modified)

. W e transfer data from individual respondent file s to the consolidated data set using automated data uploads

. For all but the simplest data transfer, 5 t his step entails transferring electronic data from the original source (e.g., Microsoft Excel spreadsheet) into the appropriate database tables using custom ized

i mport programs

. Using customized programs ensure s consistency and reproducibility, and minimize s the potential for transmission errors. We develop and test the data transfer program using a sample questionnaire to ensure that it provides error

- free transcr iption of the raw data, as verified by checking each and every imported data against the original source and review ing the error log (e.g., error automatically generated by Access when encountering a data type mismatch such as importing text into a

numeric field ) . We adjust the import program if needed

, and re

- test and verify the transfer until
100 percent of the test data are correctly uploaded into the final data set and no errors are generated

. We review the log for each individual response file imported into the database to verify that no errors were generated

. Two important considerations for transferring data include:

 Addressing potential d ata type mismatch : W e account for cases in wh ich respondents may provide data in different formats ( e.g., number f or actual values or text to indicate that the field is not applicable ) by either mapping the data to a text

- formatted field, which is the most flexible data type

, or flag ging the data point in the log for manual review.

 Differen tiating between blank s and z eros : W e translate any blank response into Null whereas we

store zero values as 0. We further evaluate Null data during QA review (see below) to identify

responses requiring follow

- up to fill data gaps versus cases in wh ich the respondent had a valid

reaso n for leaving the field blank (i.e., circumstances did not apply)

.

5 A simple data transf

er not involving the use of a customized program includes transfer from one file or table to a single database table having the same format and fields. Such simple data transfers are not common for industry surveys for which we need to combine answers from multiple respondents and several different types
of data.

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 64 

### 4.1.5 Assessment and Response Actions
We assess the processes and data on an ongoing basis throughout the data gathering phase to ensure that the primary data collection effort will meet the desired objectives. In the early phases, before the survey is mailed to respondents, response actions i nvolve revising draft documents to address comments or reviews. During the response period for the survey, any issue identified with the questionnaire design (e.g., multiple respondents contacting the hotline with similar questions) prompts discussion with the EPA WAM on the best way to address the issue and ensure smooth and accurate data collection. This may involve developing a “Frequently Asked Questions” ( FAQ ) document with clarification on questions, releasing an alternate version of the survey, or ot her appropriate response actions agreed to
with the EPA WAM.
Once survey responses come in, we conduct a preliminary review to detect any systematic issue with the data that may require adjustments to the data compilation procedures. If we identify proble ms, we quickly report them to the EPA WAM, along with the implications for uses of the data and recommended response actions.
Upon completing the transfer of individual survey responses to the database, we check that the number of

records matches the num ber of survey responses processed

, and again look for trends in the data completion rates to highlight any question that respondents may have systematically misunderstood or skipped. If we identify problems, we quickly report the issue to the EPA WAM, alon g with the implications for uses of the data and recommended response actions.



### 4.1.6 Reports to Management
We track progress in uploading and reviewing responses using the tracking database

, and provide regular updates to EPA on the number of responses that have been received and processed . As described above, once the individual responses have been compiled in to the database, we summarize the types of error s and error rate for different data elements , including missing data . We also highlight any other issues with
eventual use of the data , such respondents returning surveys indicating that they should not have had to respond , which may require recalculation of the sample weights

.



### 4.2 Data Quality Assessment and Use

This section describe s considerations for reviewing primary data prior to use in conducting analyses , but after we have obtained and compiled the primary data.



### 4.2.1 Data Review, Verification, and Validation
This stage involves a more detailed review of the data (building on the preliminary review conducted during the earlier stage to verify that the data gathered in the database match the original source s). A first
step involves going through Null entries to differentiate between valid cases (did not apply to the respondent) vers us requires follow up with the respondent

. 6 A second step involves going through the comment fields to catch any additional insight on the data.

At this sta ge, we also standardize records, verify equalities, and detect outliers through additional steps
th at include :

 Verify ing information across data elements . This first level of review seeks to verify that the information is consistent across data elements for a given respondent. This includes checking that

6 We seek to minimize the frequency of this case through survey instrument design that allows for the respondent to enter N/A when the question is not applicable.

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 65 expected relationships and equalities are met. Fo r example, the sum of generation by fuel type should equal total reported generation

.  Verify ing information across sections . This second level of review seeks to verify that the

information is consistent or not inconsistent across sections for a given resp ondent. For example, a plant that reported no electricity generation in one year in the section that addresses economic activity, should also report no revenue from sale of electricity in the section that addresses
sources of revenue

.  Verify ing data acros s the survey population . This third level of review seeks to detect discrepancies by comparing responses across respondents. This entail s comparing responses that should be the same (e.g., responses for two plants owned by the same corporate entity should report the same revenue for that entity)

. It also entails summarizing and compar ing responses across respondents to identify apparent outliers (outside the interquartile range) in the reported data or in derived metrics such as revenue per unit of a ctivity (e.g., identify cases in which a plant has unit revenue widely different from that for other plants).

 Validating data against information reported in public records

. This fourth level of review compares the survey responses to data available publi cly, when possible. Since the purpose of industry survey is to collect data that are not available from existing public data, this comparison is generally possible for select data items only or for industry totals (e.g., comparing plant

- level electricity g eneration values to data reported by EIA, comparing industry revenue to Economic Census values, comparing revenue of publicly traded companies to U.S. Securities and Exchange Commission (SEC) Form 10

- K , which are generally available from the compan y websit es ).
If our review highlights issues with a n individual respondent’s information (e.g., values provided are an order of magnitude different from those reported for other plants, potentially suggesting that the units of

measure are not those indicated), we contact the respondent to verify the data. We also follow up on incomplete data, comments that are unclear , and other issues we identified during the review. In some instances we obtain data from public records such as SEC filings. We do not overwrite ori ginal data during the QA review and validation, but instead record any revisions and final values in separate fields, along with the reason for the change.



### 4.2.2 Reconciliation with User Requirements
A final stage in the assessment is to review the survey data, after the review and validation steps have been completed, to assess if the data will meet the analytic needs or if additional steps will be required to provide the needed inputs. This assessment focuses on the data elements that are absolutely needed to

s upport the analyses.

When select survey data are not available for a given plant or owner and no additional information can be obtained from the respondent, Abt Associates will discuss with the EPA WAM appropriate methods for imputing needed values based on available data from the survey or from secondary data sources discussed

in Section 3

. For example, for Steam Electric, there were too many gaps and consistency issues in the plant revenue data reported in the survey to use the data directly. Instead, we used EIA data to imput e plant

- level revenue based on electricity sales (consistently across all plants)

, and used the survey data to
validate the EIA

- based estimates

. Other d ata imputation methods include using average values of data reported for other plants or owners (e.g., u sing average fuel costs for other plants within the same electric power market region) or estimating missing values based on other information available for the plant or owner (e.g., allocating revenue to the plant in proportion to the quantity of electric ity generated).
We document our assessment of the survey database in a memorandum to the EPA WAM, noting any concerns with the completeness or representativeness of the data. This includes documenting incomplete

### Primary Data Collection

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 66 and missing survey responses

, and responses indicating that a plant should not have been included in the sample frame (e.g., no longer in business), so that the EPA statistics team can recalculate the survey weights as needed.
Abt Associates may also need to calculate different survey weights to s upport costs and/or benefit analyses because sample weights that are based only on facility

- specific (e.g., engineering) characteristics
and various non

- facility factors can lead to a conditional bias in the estimation of national

- level benefits.

For exam ple, important non

- facility factors that influence the occurrence and size of benefits include characteristics of the receiving waters, number of regulated facilities discha rging to the affected waterbody

, and benefitting population. Abt Associates will de velop a post

- stratification weight

adjustment method (e.g., raking ) to account for additional variables that were not accounted for in the original sample design and that may affect benefit occurrenc e. We would describe these methods in
technical memoranda to the EPA WAM, and in the technical support documents for the rulemaking (e.g., Benefits Analysis Report)

.



### 4.3 Telephone Interviews

Abt Associates uses telephone interviews to collect information from small number of respondents. Such
interviews are conducte d, for example, to gather prices from equipment vendors or laboratories to support estimates of compliance costs

, or to understand certain aspects of a study or process

. Steps involved in conducting the interviews include:

 Identifying the needs for the inf ormation and the target respondents

. In this first step, we work
with the EPA WAM to establish the primary data needs based on the analytic blueprint and review of the secondary data , we also identify the entities that may be able to provide information

us ing web searches, professional contacts, etc. , agree on a number of contacts (not to exceed

nine ) that will be made to collect the information , and process for contacting the respondents (e.g., email to screen and schedule interviews)

. The number of contac ts will be based on the expected variability in the information provided; we generally consider three estimates to be appropriate for stock items.

 Developing the list of questions/interview guide

. When looking for very simple data such as the

cost s of sta ndard item s (e.g., laboratory analysis following a standard method), the interview guide consists of a simple script that introduces the interviewer, purpose for the call, and describes the request. For more complex data collection where questions may evol ve depend ing on initial answers, the interview guide includes potential follow up and scenarios.

 Gathering data. Upon approval by the EPA WAM of the target respondents and interview

process , the interviewer contacts each of the target respondent s and collects the information following the script established in the interview guide. We record the answers along with the date, time of the interview and contact information.

 Summarizing the information

. In this final step, we prepare a memorandum compil ing the data across the interviews, presenting the distribution of values and recommendations for the use of the data in the analyses

.

### References

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 67 

### 5. References

Bateman, I.J. and A.P. Jones. 2003. Contrasting Conventional with Multi

- Level Modeling Approaches to

Meta

- Analysis: Expectation Consistency in U.K. Woodland Recreation Values. Land Economics 79(2):
235

- 258.
Bergstrom, J.C. and P. De Civita. 1999. Status o f Benefits Transfer in the United States and Canada: A Review. Canadian Journal of Agricultural Economics 47(1): 79

- 87.
Bergstrom, J.C. and L.O. Taylor. 2006. “Using Meta

- Analysis for Benefits Transfer: Theory and Practice.” Ecological Economics 60(2):351

- 360.
Boyle, K. J., Parmeter, C. F., Boehlert, B. B., & Paterson, R. W. 2013. Due diligence in meta

- analyses to

s upport benefit transfers. Environmental and Resource Economics, 55(3), 357

- 386.
California Environmental Protection Agency, Office of Environm ental Health Haza rd Assessment (Cal OEHHA). 2013. Legget Model Code. Available at: http://oehha.ca.gov/air/legget.html

. Dalhuisen, J.M., R.J.G.M. Florax, H.L.F. de Groot, and P. Nijkamp. 2003. Price and i ncome elasticities of residential water demand: A meta analysis. Land Economics 79(2):292

- 308.
Dillman, D.A., Smyth, J.D. and Christian, L.M. 2009. Internet, Mail, and Mixed Mode Surveys: The Tailored Design Method (3 rd ed.). New York, NY: John Wiley and Sons.
Glass, G.V. 1976. Primary, Secondary, and Meta

- Analysis of Research. Educational Researcher 5(10): 3

- 8.
Johnston, R.J., M.H. Ranson, E.Y. Besedin and E.C. Helm. 2006. “What Determines Willingness To Pay

Per Fish? A Meta

- Analysis of Recreational Fish ing Values.” Marine Resource Economics 21:1

- 32.

Johnston, R.J., E.Y. Besedin, R. Iovanna, C.J. Miller, R.F. Wardwell and M.H. Ranson. 2005. “Systematic Variation in Willingness to Pay for Aquatic Resource Improvements and Implications for

Benefit Transfer : A Meta

- Analysis.” Canadian Journal of Agricultural Economics 53(2

- 3):221

- 248.
Johnston, R.J., J.J. Opaluch, T.A. Grigalunas, and M.J. Mazzotta. 2001. Estimating Amenity Benefits of Coastal Farmland. Growth and Change 32(summer): 305

- 325.

Leggett R

. W

. 1 993. An age

- specific kinetic model of lead metabolism in humans. Environ Health Perspect 101(7): 598

- 616.
Moeltner, K., K.J. Boyle, and R.W. Paterson. 2007. “Meta

- Analysis and Benefit Transfer for Resource

Valuation

— Addressing Classical Challenges with Bay esian Modeling.” Journal of Environmental Economics and Management 53(2):250

- 269.
Navrud, S. and R. Ready. 2007. Review of Methods for Value Transfer. In S. Navrud and R. Ready (eds.), Environmental Value Transfer: Issues and Methods. Dordrecht, The Nethe rlands: Springer. pp.1

- 10.
Poe, G.L., K.J. Boyle, and J.C. Bergstrom. 2001. A Preliminary Meta Analysis of Contingent Values for Ground Water Quality Revisited, in J.C. Bergstrom, K.J. Boyle and G.L. Poe, eds. The Economic Value of Water Quality. Northamp ton, MA: Edward Elgar Publishers.
Rosenberger, R.S. and R.J. Johnston. 2007. Selection Effects in Meta

- Valuation Function Transfers.

Meta

- Analysis in Economics Conference, September 27

- 30, 2007, Sandbjerg Manor, Sønderborg,
Denmark.
Rosenberger, R.S. and J.B. Loomis. 2000. Using Meta

- Analysis for Benefit Transfer: In

- Sample Convergent Validity Tests of an Outdoor Recreation Database. Water Resources Research 36(4): 1097

- 1107.

### References

Abt Associates PQAPP EP

- C

- 13

- 039 July 2014 68 Santos, J.M.L. 1998. The Economic Valuation of Landscape Change. Cheltenham, UK : Edward Elgar.
Smith, V.K., and L. Osborne. 1996. Do Contingent Valuation Estimates Pass the Scope Test? A Meta Analysis. Journal of Environmental Economics and Management 31(3): 287

- 301.
Stanley, T. D., Doucouliagos, H., Giles, M., Heckemeyer, J. H., J o hnston, R. J., Laroche, P., & Rost, K.
2013 . Meta

- analysis of economics research reporting guidelines. Journal of Economic Surveys, 27(2),

390

- 394.
United States Energy Information Administration (EIA). 2009. Form EIA

- 860 Detailed Data.
United States Environmental Protection Agency (EPA). December 2010. Guidelines for Preparing Economic Analyses.
United States Environmental Protection Agency (EPA). July 2010. EPA’s Action Development Process: Interim Guidance on Considering Environmental Justice During the Development of an Action.
United States Environmental Protection Agency (EPA). October 2009. Technical Support Document for the Preliminary 2010 Effluent Guidelines Program Plan. EPA 821

- R

- 09

- 006

. United States Environmental Protection Agency (EPA). November 2006. EPA’s Action Development Process. Final Guidance for EPA Rulewriters: Regulatory Flexibility Act.
United States Environmental Protection Agency (EPA). 2006. Peer Review Handbook, 3 rd Edit ion. Prepared for EPA’s Science Policy Council by Member of the Peer Review Advisory Group. EPA/100/B

06/002.
United States Environmental Protection Agency (EPA). 2002. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity , of Information Disseminated by the Environmental Protection Agency. EPA/260R

- 02

- 008.

### Appendix A

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 69 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. 

### Appendix A : Checklists

Table A

- 1 : Document QA /QC Checklist

General Information

Work Assignment Number:

Deliverable:
Review Level/Reason (including quick turnaround):

Work Assignment Manager :
Project Staff:
Project Quality Assurance Advisor:

Due date:

Document QA

/QC Checklist Place an "x" in the appropriate check box followed by the name of the reviewer and date reviewed.

Level 1 Yes No N/A Reviewer Date Comments Outline reviewed by WAM (and PQA Advisor if appropriate).
Detailed review by author and peer (or WAM).
Cited numbers match underlying sources.

Principal sources cited; references included.

Notes principal limitations.
Describes QA activities, per QAPP.
Document is free of grammatical errors.
Document uses established formatting style.
Level 1 Completed:

### Appendix A

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 70 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Document QA

/QC Checklist Place an "x" in the appropriate check box followed by the name of the reviewer and date reviewed.
Level 2 (includes Level 1) Yes No N/A Reviewer Date Comments Detailed review by WAM for accuracy, clarity of presentation, and completeness. Detailed review by PQA Advisor or his/her designee, including for compliance with QAPP.

All sources cited; references included. Discusses limitations, biases, and uncertainties in separate section.
Optional review by editor.
Optional formatting review.
Level 2 Completed:

Level 3 (includes Levels 1 and 2) Yes No N/A Reviewer Date Notes
Editorial and formatting review (clarity of presentation, consistency of formatting, etc.).
Final acceptance by Project Manager or his/her designee, including verifying use of appropriate language and characterization of the results. Level 3 Completed:

### Appendix A

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 71 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Table A

- 2 : Spreadsheet QA/QC Checklist

General Information

Work Assignment Number:

Deliverable:
Review Level/Reason (including quick turnaround):

Work Assignment Manager :
Project Staff:
Project Quality Assurance Advisor:

Due date:

Spreadsheet QA

/QC Checklist

Add additional rows for additional sheets when necessary. Place an "x" in the appropriate check box followed by the reviewer and date reviewed.

File General Yes No N/A Reviewer Date Comments Formulas/calculations correctly reflect general methodology and assumptions Results have been replicated/ reproduced by external calculation. Provide information on software and methodology used. Results follow expected direction and approximate magnitude of change across analyzed cases (e.g., regulatory options) References are appropriately cited Assumptions, inputs, data, and outputs are shown on separate sheets.
Hard

- coded inputs or inputs calculated from hard

coded values only in Assumptions or Data sheets.

Sheet 1 Yes No N/A Reviewer Date Comments Data are read in correctly/data sources are verified and documented Formulas/calculations correctly reflect general methodology and assumptions Formulas are correctly written Formulas refer to the appropriate cells Multiple steps in a calculation are shown in their own cell/row/column, to the extent practicable.

### Appendix A

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 72 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Spreadsheet QA

/QC Checklist

Add additional rows for additional sheets when necessary. Place an "x" in the appropriate check box followed by the reviewer and date reviewed.

Sheet 2 Yes No N/A Reviewer Date Notes Data are read in correctly/data sources are verified and documented Formulas/calculations correctly reflect general methodology and assumptions Formulas are correctly written Formulas refer to the appropriate cells Multiple steps in a calculation are shown in their own cell/row/column, to the extent practicable.

Table A

- 3 : Programming/Modeling QA/QC Checklist

General Information

Work Assignment Number:

Deliverable:

Review Level/Reason (including quick turnaround):

Work Assignment Manager :
Project Staff:
Project Quality Assurance Advisor:

Due date:

Programming/Modeling QA /QC Checklist Place an "x" in the appropriate check box followed by the reviewer and date reviewed.

Yes No N/A Reviewer Date Comments Program has proper documentation or a "read me" file which adequately explains data sources, program functions, and outputs. Data are read in correctly. Program code is written correctly and contains no

errors. Overall program process is consistent with the results produced Results can be independently replicated/reproduced with another program.

### Appendix B

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 73 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. 

### Appendix B : QA Audit Checklist

Project Name: Work Assignment: Work Assignment Manager:

Auditor:

Date:

Instructions : 1. Obtain the project work plan, QAPP, and any other planning documents from the Project Manager. 2. Review the QA requirements. 3. Interview the Project Manager and other team members, if necessary.



### B.1 Planning and Project Management
Quantitative Review (to be completed by auditor after review of project files or discussion with appropriate team members)

1. What planning documents are available for this project?
☐ QAPP
☐ Work Plan

☐ Project Instructions ☐ Required by client
☐ Required by client

☐ Required by client ☐ Provided to client
☐ Provided to client

☐ Provided to client ☐ Reviewed by team
☐ Reviewed by team

☐ Reviewed by team

2. Were there any changes to the QAPP during the project, and if so, did the WAM distribute the changes to the project team and client?
☐ Yes ☐ No ☐ N/A

3. D id the client request any QA reports? If so

, did we provide them?
☐ Yes ☐ No ☐ N/A

4. Did the project team, PQA, or client identify any quality deficiencies during the project? If so, specify.
☐ Yes ☐ No ☐ N/A



### B.1 QA Spe cifications

1. Are there QA specifications for the project in the QAPP, Work Plan, or other key document? (Check all that apply)
☐ Conceptual review (Complete Question 2) ☐ Calculation review (Complete Question 3)

☐ Deliverable review (Complete Question 3) ☐ Senior review (Complete Question 6)
☐ Peer review (Complete Question 6) ☐ Other review (specify):

### Appendix B

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 74 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Conceptual Review

2. Was there a conceptual review for planning major activities/deliverables? ☐ Yes ☐ No If y es, does the reviewer have enough experience and a relevant degree? ☐ Yes ☐ No Typically conceptual reviewers are at least at Professional Level 3, and often are the project manager or senior reviewer. Ask the project manager if there are other reviewer qualifications specified (e.g., someone outside the project, professional engineer, P rofessional Level 4, etc.).

Developmental Review
3. How are calculation and deliverable reviews documented? Sign

- off Sheet ☐ Yes, always ☐ Yes, mostly ☐ Sometimes ☐ No Tracking Database ☐ Yes, always ☐ Yes, mostly ☐ Sometimes ☐ No Other, explain:

4. Do subcontractors prepare deliverables?
☐ Yes ☐ No ☐ N/A

5. If yes to question 4, what is the review process for their work products?

Senior and Peer Review

6. Was there a senior or peer review of major activities/deliverables?
☐ Yes ☐ No ☐ N/A

7. If yes to question 6 , who is the reviewer? Does the reviewer have enough experience and a relevant degree? Typically senior and peer r eviewers are at least P rofessional Level 3. Ask the project manager if there are other reviewer qualifications specified (e.g., someone outside the project, professional engineer,

P rofessional Level 4, etc.).

Obstacles to Review Process 8. According to the PM or other team members, what were the obstacles to performing any level of
review?

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 75 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. 

### Appendix C : Meta

- Analysis Data and Methods

One of the benefits transfer approaches that Abt Associates employs in supporting EAD in developing
ELGs is meta

- analysis. Abt Associates is developing two meta

- analyses: (1) surface water quality valuation studies and (2) hedonic property studies of open space values. This appendix describes the data gathering (including specific sources), assessment, and use pertaining to estimating the meta

- regression

mode ls, and reporting the resulting values.

The meta

- analysis of surface water quality valuation studies is a re

- examination and update of an existing

meta

- analysis that Abt Associates developed for EPA. While the existing meta

- analysis model h as been

peer rev iewed

, and has been useful for cost

- benefit analyses for multiple rulemakings, the reexamination is necessary to incorporate recent advances in meta

- analysis, address limitation s of the existing models (i.e., the inability to capture the effects of geograp hic scale of affected resources and the availability of regional substitutes ) and improve the model for future rulemaking support. The reexamination includes a literature review to identify additional studies to include in the meta

- data

, and deve lopment an d compilation of additional variables to include in a revised WTP function with a focus on geographic characteristics of affected resources and benefitting populations ( e.g., geographic area covered by the study, size of the affected aquatic resource ). Rev isions to this P QAPP may be necessary once re

examination is complete

, and there has be e n peer review of the revised meta

- analysi s

.



### C.1 Identifying Candidate Studies
Following the literature search procedures described in the body of this PQAPP (see Section 3.1.2 ), we
will search for stated or revealed preference studies (e.g., hedonic implicit price models) us ing the following specific terms to identi fy valuation studies of surface water quality, and hedonic studies of open space values , to be sufficiently broad so not to miss relevant studies:

 Water quality, water clarity, nutrient removal/improvement, water quality index/ladder, clean
water, water p ollution reduction, habitat improvement, stream flow

 O pen space, low impact development ( LID ) , BMP, green infrastructure, conservation subdivision, environmental site design ( ESD ) , property value

, property price, WTP, hedonic, non

market values, residentia l subdivision, homeowner preferences

. In compiling meta

- data, we are following Stanley et al. (2013) guidelines for documenting and reporting

meta

- regression analyses in economics , including document ation of the new literature search . This

documentation i ncludes: (a) the exact databases and other sources searched, (b) the precise combination of keywords, and (c) date completed.

To be included in the metadata, candidate studies must be both commodity consistent and welfare consistent (Bergstrom and Taylor 2006). For commodity consistency, the amenity valued should be similar across studies in the metadata. Studies must also provide sufficient detail on the amenity itself as discussed by Johnston et al. (2005) : “the study must provide sufficient information regarding resource, context, and study attributes to warrant inclusion in the meta

- data.” Following the requ irement fo r welfare consistency, the meta data will not combine studies representing theoretically distinct welfare measures (i.e., stated and revealed preference data ).
Candidate studies for the water quality and hedonic meta

- analyses must also meet criteria for study location and quality. All studies must be conducted in the U nited S tates or Canada

, and must apply
research methods th at are supported by the literature and that provide WTP estimates consistent with neoclassical welfare theory.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 76 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. We will also take steps to identify duplicative studies. Some studies may be released in multiple forms, such as working papers, conference paper s, journal articles, and book chapters. We will screen articles to remove duplicative analysis and analys e s that were subsequently revised and published in peer reviewed journals

. Multiple values can be included for a study if they address a different com ponent of the study
data (e.g., a different watershed or affected population) or estimate values for alternative resource changes.

Sections C.4 and C.5 list the data sources we identified from these searches that meet the screening criteria described in the body of this PQAPP (see Section 3.1.3 ).



### C.2 Estimating the Models
Glass (1976) characterizes meta

- analysis as “the statistical analysis of a large collection of results for

individual studies for the purposes of integrating the findings. ” Meta

- analysis is largely an empirical,

data

- driven process, but one in which variable and model selection is guided by theory (Bergstrom and Taylor 2006). Given a reliance on information available fro m the underlying studies that comprise the metadata, meta

- analysis models most often represent a middle ground between model specifications that would be most theoretically appropriate and those specifications that are possible given available data.

Abt A ssociates follows the mechanics of specifying and estimating meta

- equations in resource economics applications as outlined in Smith and Osborne (1996), Rosenberger and Loomis (2000), Poe et al. (2001), Bateman and Jones (2003), Dalhuisen et al. (2003), Joh nston et al. (2005, 2006), Bergstrom and Taylor (2006), Moeltner et al. (2007). More specifically, Abt Associates follows work by Bateman and Jones (2003), and Johnston et al. (2005), in applying a multilevel model specification to metadata, to address
pot ential correlation among observations gathered from single studies. Also following prior work (e.g., Smith and Osborne 1996; Poe et al. 2001), we apply the Huber

- White robust variance estimation. As described by Smith and Osborne (1996, p. 293), “this appr oach treats each study as the equivalent of a sample cluster with the potential for heteroskedasticity...across clusters.”
To guide development of the model and variable specifications, we rely upon a set of general principles designed to help prevent exces sive data manipulations and other factors that may lead to misleading
model results. All else being equal, t he general principles include :

 Fewer and simpler data transformations are better than more extensive transformations

 In the absence of overriding t heoretical considerations, continuous variables are generally better than discrete variables derived from underlying continuous distributions

 M odels should capture elements of the scope and scale of resource changes (e.g., the size of the
affected waterbod ies) as well as availability of substitute sites

 Models should distinguish WTP associated with different types of resources and resource uses, particularly when relevant to the policy question at hand

 Following the “weak structural utility theoretic” (WS UT) approach of Bergstrom and Taylor (2006, p. 352), we avoid exogenous structural constraints to afford the flexibility necessary to appropriately model empirical patterns that may not necessarily flow from an underlying theoretical modeling structure.

In the case of the meta

- analysis of surface water valuation studies, the revised model is being specified in order to capture four primary types of scope and scale effects on WTP, including the extent of the market analyzed by each primary study. The potenti al relevance of these effects is supported by theory and empirical findings in the primary valuation literature:

1. Scope of quality change (the ma gnitude of water quality change )

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 77 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. 2. The geographical scale over which change occurs (the size of the water bodies a ffected by the change)
3. Extent of the market (the size of the sampled area, or geographic area over which WTP is quantified)
4. The proportion of affected versus unaffected resources (the size of affected water bodies relative
to all water bodies in the sampled market) We will develop scope and scale variables that are meaningful, measurable, and unambiguous.

For each meta

- analysis, we may estimate different model specifications and select the final model specifications using the following criteria: (1) fit, (2) intuitive results provided (including sign, statistical significance and correlations) , and (3) common use in the empirical valuation and meta

- analysis literature (e.g., Smith and Osborne 1996; Santos 1998; Johnston et al. 2001, 2005, 2006). Mode ls are also evaluated based on correspondence of specifications to theory and prior expectations.

To test robustness of the parameter estimates from the model, we will conduct a set of procedures suggested by Boyle, et al. (2013), including tests for sample selection bias, horizontal robustness (inclusion/exclusion of observations or studies), and vertical robustness (inclusion/exclusion of variables). It is possible that the literature does not include studies for some areas of the country and, as a r esult, the metadata could have better coverage for some areas of the country than others . Evaluation of geographic representativeness will be included mapping the location of st udies included in the metadata. To assess potential sample selection bias we wi ll rely on “due diligence” tests discussed by Boyle et al. (2013).



### C.3 Documenting, Interpreting, and Presenting Results
We fully document the variables that we include and exclude in the regression models, along with the rationale. For example, we may exclude variable that are not statistically significant in any version of the model and for which there is no overriding rationale for retaining the variable in the model. For example, a variable distinguishing different types of discrete choice instruments (e.g. , conjoint versus dichotomous choice) may add no significant explanatory power to the model. In comparison, we may retain certain variables for theoretical reasons, despite in significance. Such specification of meta

- analysis models using
a combination of t heoretical guidance and empirical considerations is standard in modeling efforts (Bergstrom and Taylor, 2006).
We also fully document our results, including variable; parameter estimate; and standard error. We review measures of equation fit, that the sign s of significant parameter estimates correspond with intuition, and study level covariance factors. We may also compare our results to prior meta

- analyses in the valuation
literature.
One of the primary means to assess the validity of benefit transfers

— an d the only one that may be applied in cases wherein the true value for the study site is unknown

— is a value surface test (Bergstrom and De Civita, 1999). These tests involve assessments of ways in which “different factors may cause values to vary across si tes, providing guidance for adjustments needed to make a valid transfer of value estimates from the study site(s) to the policy site” (Bergstrom and De Civita, 1999). We follow general approaches for such value surface assessments, which generally involve comparisons of empirical patterns found via

meta

- analysis to theoretical expectations or norms.

Results of meta

- analysis are dependent on the sample of studies available for the given resource change (Navrud and Ready, 2007), and may be subject to various selection biases if the available literature does not provide a representative, unbiased perspective on welfare estimates associated with resource changes (Rosenberger and Johnston, 2007). However, we also take steps to ameliorate such potential biases,

in cluding the incorporation of both peer

- reviewed and gray literature to avoid possible publication biases (Rosenberger and Johnston, 2007), and the use of a comprehensive literature review in the attempt to

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 78 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. avoid

— as much as possible

— other types of selection biases. We also note when results should be viewed
with caution.
The resulting preferred meta

- analysis models will be peer reviewed according to EPA’s Peer Review Handbook (EPA 2006). Peer review is an important component of EAD’s quality control program

, providing a “process for enhancing a scientific or technical work product so that the decision or position taken by the Agency, based on that product has a sound, credible basi s” ( EPA 2006). The burden for implementing the peer review rests on EAD, not Ab t Associates.



### C.4 : Studies Used in Surface Water Valuation Meta

- Analysis
Abt Associates is using the following studies in meta

- analysis of surface water values, and will update this PQAPP with additional results of ongoing efforts as needed.

Adamowicz, W.L., Louviere, J., and Williams, M. 1994, Combining Revealed and States Preference Methods for Valuing Environmental Amenities. Journal of Environmental Economics and Management. Vol (26), pp. 271

- 292, 1994.

Aiken, Richard A. 1985. Public Benefits of Envi ronmental Protection in Colorado. Masters thesis, Colorado State University.
Anderson, G.D. and S.F. Edwards. 1986. Protecting Rhode Island’s Coastal Salt Ponds: An Economic Assessment of Downzoning. Coastal Zone Management Journal. 14( 1⁄2): 67

- 91.

Aze vedo, Christopher, J.A. Herriges, and C.L. Kling. 2001. Valuing Preservation and Improvements of Water Quality in Clear Lake. Center for Agricultural and Rural Development (CARD), Iowa State University, Staff Report 01

- SR 94.
Banzhaf, Spencer H, Dallas Burtraw, David Evans, and Alan Krupnick. 2006. "Valuation of Natural Resource Improvements in the Adirondacks." Land Economics, 82(3): 445

- 464.
Bockstael, N.E., K.E. McConnell and I.E. Strand. 1989. Measuring the Benefits of Improvements in Water Quality : The Chesapeake Bay. Marine Resource Economics. Vol. 6: 1

- 18.
Bockstael, N.E., McConnell, K.E., and Strand, I.E. 1988. Benefits from improvements in Chesapeake Bay Water Quality. Department of Agricultural & Resource Economics, University of Maryland, 1988.
Borisova, Tatiana, Alan Collins, Gerard D’Souza, Matthew Benson, Mary Leigh Wolfe, and Brian Benham, 2008. A Benefit

- Cost Analysis of Total Maximum Daily Load Implementation. Journal of the American Water Resources Association (JAWRA) 44(4):1009

- 1023 . DOI: 10.1111/j.1752

- 1688.2008.00216.x
Breffle, William S., Morey, E., Rowe, R., Waldman, D. and Wytinck, S. 1999. Recreational Fishing Damages from Fish Consumption Advisories in the Waters of Green Bay. Prepared by Stratus Consulting Inc. for US Fish and Wildlife Service, US Department of Interior and US Department of Justice, 1999.
Brox, J., R. Kumar and K.R. Stollery. 2003. Estimating Willingness to Pay for Improved Water Quality in the Presence of Item Nonresponse Bias. American Journal of Agricult ural Economics, Vol (85), pp.

414

- 428, 2003.
Cameron, T.A. and D.D. Huppert. 1989. OLS versus ML Estimation of Non

- market Resource Values with Payment Card Interval Data. Journal of Environmental Economics and Management. 17, 230

- 246.
Cameron, Trudy Ann. 1988. Using the Basic "Auto

- validation" Model to Assess the Effect of Environmental Quality on Texas Recreational Fishing Demand: Welfare Estimates. EPA National Center for Environmental Economics, 1988.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 79 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Carson, R., Mitchell, R., Hanemann, W. M., Kopp, R ., Presser, S., Ruud, P. 1992. A Contingent Valuation Study of Lost Passive Use Values Resulting from the Exxon Valdez Oil Spill. A Report to the Attorney General of the State of Alaska, 1992.

Carson, Richard T., & Mitchell, Robert C, "The Value of Clea n Water: The Public's Willingness to Pay for Boatable, Fishable, and Swimmable Quality Water," Water ResourcesResearch, vol. 29, 1993.
Carson, Richard T., W. M. Hanemann, R. J. Kopp, J. A. Krosnick, R. C. Mitchell, S. Presser, P. A. Ruud, and V. K. Smith. 1994. Prospective Interim Lost Use Value due to DDT and PCB Contamination in the Southern California Bight. Volume 2.

Clonts, Howar d A., and J.W. Malone. 1990. Preservation attitudes and consumer surplus in free flowing rivers. In: Social Science and Natural Resource Recreation Management, Joanne Vining, editor. Westview Press, Boulder, CO. Vol. pp. 301

- 317.

Collins, A. R., R. S. Rosenberger, et al. (2009). Valuing the restoration of acidic streams in the Appalachian Region: A stated choice method. Environmental economics for watershed restoration. H. W. Thurston, M. T. Heberling and A. Schrecongost. Boca Raton, CRC Press.

Collins , Alan R. and Rosenberger, Randall S., (2007), Protest Adjustments in the Valuation of Watershed Restoration Using Payment Card Data, Agricultural and Resource Economics Review, 36, issue 2.
Corrigan, J R, K J Egan, and J A Downing. 2009. "Aesthetic Values of Lakes and Rivers." Elsevier Inc.

Croke, Kevin, R.G. Fabian, and G. Brenniman. 1986

- 87. Estimating the Value of Improved Water Quality in an Urban River System. Journal of Environmental Systems. Vol. 16, No. 1. pp. 13

- 24.

Cronin, Francis J. 1982. Valuing nonmarket goods through contingent markets. Pacific Northwest Laboratory, PNL 4255, Richland, WA.
De Zoysa, A. D. N. 1995. A Benefit Evaluation of Programs to Enhance Groundwater Quality, Surface Water Quality and Wetland Habitat in Northwest O hio. Dissertation, Ohio State University.
Desvousges, William H., Smith, V. Kerry and Fisher, Ann. 1987. Option Price Estimates for Water Quality Improvements: A Contingent Valuation Study for the Monongahela River. Journal of
Environmental Economics an d Management. Vol (14), pp. 248

- 267, 1987.
Desvousges, William H., V.K. Smith, and M.P. McGivney. 1983. A Comparison of Alternative Approaches for Estimating Recreation and Related Benefits of Water Quality Improvements. U.S.
Environmental Protection A gency, Economic Analysis Division, Washington.
Douglas, A. J. and Sleeper, A. 2005. Estimating Recreation Trip Related Benefits for the Klamath River Basin with TCM and Contingent Use Data.
Egan, K., Herriges, J., Kling, C. and Downing, J. 2004. Recreat ion Demand Using Physical Measures of Water Quality. Working Paper, Center for Agricultural and Rural Development, Iowa State University,

2004.
Farber, S., and Griner, B. (2000). “Using Conjoint Analysis to Value Ecosystem. Change,”

Environmental Science and Technology, 34(8), 1407

- 1412.
Farber, Stephen, and B. Griner. 2000. Valuing watershed quality improvements using conjoint analysis. Ecological Economics. 34 (1): 63

- 76.

Feather, P.M., Hellerstein, D. and Tomasi, T. 1995. A Discrete

- Count Model of Recreation Demand. Journal of Environmental Economics and Management, Vol (29), pp. 214

- 227, 1995.
Greene, G., Moss, C. and Spreen, T. 1997. Demand for Recreational Fishing in Tampa Bay, Florida: A Random Utility Approach. Marine Resource Economics, Vo l (12), pp. 293

- 305, 1997.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 80 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Greenley, D.A., R.G. Walsh, and R.A. Young. 1981. Option Value: Empirical Evidence from a Case Study of Recreation and Water Quality. The Quarterly Journal of Economics, Vol (96), pp. 657

- 674,
1981.
Hayes, K.M., T.J.Tyrell and G. Anderson. 1992. Estimating the Benefits of Water Quality Improvements in the Upper Narragansett Bay. Marine Resource Economics. Vol. 7, pp. 75

- 85.
Herriges, Joseph A., and J.F. Shogren. 1996. Starting Point Bias in Dichotomous Choice Valuation with

F ollow

- Up Questioning. Journal of Environmental Economics and Management. Vol. 30, No. 1, pp.

112

- 131.
Hite, D. 2002. Willingness to Pay for Water Quality Improvements: The Case of Precision Application Technology. Deaprtment of Agricultural Economics and Rural Sociology, Auburn University, Auburn, AL; 8 August 2002.
Hoehn, J., Tomasi, T., Lupi, F., and Chen, H. 1996. An Economic Model for Valuing Recreational Angling Resources in Michigan. Volume I. Report submitted to Michigan Department of
Environme ntal Quality and Michigan Department of Natural Resources, 1996.
Holmes, Thomas P. & Bergstrom, John C. & Huszar, Eric & Kask, Susan B. & Orr, Fritz III, 2004. "Contingent valuation, net marginal benefits, and the scale of riparian ecosystem restoration," Ecological Economics, Elsevier, vol. 49(1), pages 19

- 30, May.

Huang, Ju

- Chin, T. Haab, J.C. Whitehead. 1997. Willingness to Pay for Quality Improvements: Should
Revealed and Stated Preference Data Be Combined? Journal of Environmental Economics and

Man agement. Vol. 34, No. 3, pp. 240

- 255.
Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU

- TB

042 2000, 1999.

Hushak, L . and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU

- TB

042 2000, 2000.
Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU

- TB

042 2000, 2001.
Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of
Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU

- TB

042 2000, 2002.
Kaoru, Yoshiaki. 1993. Differentiating Use and Nonuse Values for Coastal Pond Water Quality Improvements. Environmental and Resource Economics. Vol. 3. pp. 487 494.
Lant, C. L. and R.S. Roberts. 1990. Greenbelts in the Cornbelt: Riparian Wetlands, Intrinsic Values, and Market Failure. Environment and Planning A, Vol. 22, pages 1375

- 1388.

Lant, C.L., and G.A. Tobin. 1989. The Economic Value of Riparian Corridors in Cornbelt Floodplains: A Research Framework. Professional Geographer Vol (41), pp. 337

- 349, 1989.
Lichtkoppler, F.R. and T.W. Blaine. 1999. Environmental Awareness and Attitudes of Ashtabula County Voters Concerning the Ashtabula Rive r Area of Concern: 1996

- 1997. J. Great Lakes Resources, Vol (25), pp. 500

- 514, 1999.
Lindsey, G. 1994. Market Models, Protest Bids, and Outliers in Contingent Valuation. Journal of Water Resources Planning and Management, Vol (12), pp. 121

- 129, 1994.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 81 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Lipt on, D. 2004. The Value of Improved Water Quality to Chesapeake Bay Boaters. Marine Resource Economics. Vol(19):265

- 270.
Londoño Cadavid, C. and A. W. Ando (2013). "Valuing preferences over stormwater management outcomes including improved hydrologic functi on." Water Resources Research 49: 4114

- 4125.
Loomis, J., Paula Kent, Liz Strange, Kurt Fausch and Alan Covich. Measuring the total economic value of restoring ecosystem services in an impaired river basin: results from a contingent valuation survey.
Ecolo gical Economics, Vol (33), pp. 103

- 117, 2000.
Loomis, J., Paula Kent, Liz Strange, Kurt Fausch and Alan Covich. Measuring the total economic value of restoring ecosystem services in an impaired river basin: results from a contingent valuation survey.
Ecol ogical Economics, Vol (33), pp. 103

- 117, 2000
Loomis, John B. 1996. How Large is the Extent of the Market for Public Goods: Evidence from a Nation Wide Contingent Valuation Survey. Applied Economics. 28(7): 779

- 782.
Lyke, A.J. 1993. Discrete Choice Mo dels to Value Changes in Environmental Quality: A Great Lakes Case Study. Dissertation submitted to the Graduate School of the University of Wisconsin, Madison.
Magat, Wesley A., J. Huber, K.W. Viscusi, and J. Bell. 2000. An Iterative Choice Approach to Valuing Clean Lakes, Rivers, and Streams. Journal of Risk and Uncertainty. Vol. 21, No. 1. pp. 7

- 43.
Matthews, Leah Greden, F.R. Homans, and K.W. Easter. 1999. Reducing Phosphorous Pollution in the Minnesota River: How Much is it Worth? Staff Paper. Department of Applied Economics, University of
Minnesota.
Mazzotta, Marisa J. 1996. Measuring Public Values and Priorities for Natural Resources: An Application to the Peconic Estuary System. Ph.D. Dissertation, University of Rhode Island.

Mitchell, Robert Cameron, and R.T. Carson. 1981. An experiment in determining willingness to pay for national water quality improvements. Preliminary Draft of a report to the U.S. Environmental Protection Agency. Resources for the Future, Inc., Washin.

Morey, E. , Breffle, W., Rowe, R. and Waldman, D. 2002. Estimating Recreational Trout Fishing Damages in Montana's Clark Fork River Basin: Summary of a Natural Resource Damage Assessment. Journal of Environmental Economics and Management, 66(2), pp. 159

- 170, 2002.
Needelman, M. and Kealy, M. 1995. Recreational Swimming Benefits of New Hampshire Lake Water Quality Policies: An Application of a Repeated Discrete Choice Model. Agricultural and Resource Economics Review, Vol (24), pp. 78

- 87, 1995.

Olsen, D., J. Richa rds, and R.D. Scott. 1991. "Existence and Sport Values for Doubling the Size of Columbia River Basin Salmon and Steelhead Runs. Rivers 2(1):44

- 56.
Patrick, R., J. Fletcher, S. Lovejoy, W. Van Beek, G. Hollowy, and J. Binkley. 1991. Estimating Regional Benefits of Reducing Targeted Pollutants: An Application to Agricultural Effects on Effects on Water Quality and the Value of Recreational Fishing. Journal of Environmental Management, Vol (33), pp. 301

- 310, 1991.

Peters, T., W. L. Adamowicz and P.C. Box all. 1995. Influence of choice set considerations in modeling the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781

– 1788, 1995.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781

– 1788, 1996.
Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781

– 1788, 1997.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 82 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781

– 1788, 19 98.
Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781

– 1788, 1999.

Peters, T., W. L. Adamowicz and P.C. Boxall. 199 5. Influence of choice set considerations in modeling the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781

– 1788, 2000.
Phaneuf, D. 2002. A Random Utility Model for TMDLs: Estimating the Benefits of Watershed Based

Ambient Water Quality Improvements. Department of Agricultural and Resource Economics, North Carolina State University, 2002.
Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality Improvements Using Revealed Preference Meth ods When Corner Solutions Are Present. 80(5), Dec
1998.
Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec

1999.

Phan euf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec

2000.
Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 19 98. Valuing Water Quality Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec

2001.
Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality Improvements Using Revealed Prefere nce Methods When Corner Solutions Are Present. 80(5), Dec
2002.
Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec

2003.
Phaneuf, Daniel J., Laura O. Taylor, and John Braden. 2013. "Combining RP and SP data to estimate preferences for residential amenities: a GMM approach" Land Economics 89(1):30

- 52.

Roberts, Lisa A., J.A. Leitch. 1997. Economic Valuation of Some Wet land Outputs of Mud Lake. North Dakota State University Dept. of Agricultural Economics, Report #381.
Rowe, Robert D., W.D. Schulze, B. Hurd, and D. Orr. 1985. Economic Assessment of Damage Related to the Eagle Mine Facility. Energy and Resource Consu ltants, Inc., Boulder, CO.
Sanders, L. B., R.G. Walsh, and J.B. Loomis. 1990. Toward Empirical Estimation of the Total Value of Protecting Rivers. Water Resources Research. Vol. 26, No. 7. pp. 1345 1357.

Schulze, William D., R.D. Rowe, W.S. Breffle, R.R. Boyce, and G.H. McClelland. 1995. Contingent Valuation of Natural Resource Damages Due to Injuries to the Upper Clark Fork River Basin. State of Montana, Natural Resource Damage Litigation Program.
Shresta, R. K., J.R.R. Alavalapati. 2004. Valuing Env ironmental Benefits of Silvopasture Practice: A Case Study of the Lake Okeechobee Watershed in Florida. Ecological Economics, Vol (49), pp. 349

- 359,

2004.
Smith, V. and W. Desvousges. 1985. The Generalized Travel Cost Model and Water Quality Benefits: A

R econsideration. Southern Economic Journal, Vol (52), pp. 371

- 381, 1985.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 83 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Stumborg, Basil E., K.A. Baerenklau, and R.C.. Bishop. 2001. Nonpoint Source Pollution and Present Values: A Contingent Valuation of Lake Mendota. Review of Agricultural Economics. 23(1): 120

- 132.
Sutherland, R. J., and R.G. Walsh. 1985. Effect of Distance on the Preservation Value of Water Quality. Land Economics. Vol. 61, No. 3. pp. 282 291.
Takatsuka, Yuki, "Comparison of the Contingent Valuation Method and the Stated Choice Mod el for Measuring Benefits of Ecosystem Management: A Case Study of the Clinch River Valley, Tennessee. " PhD diss., University of Tennessee, 2004.
Tay, R. McCarthy, P. 1994. Benefits of Improved Water Quality: A Discrete Choice Analysis of
Freshwater Recre ational Demands. Environment and Planning. Vol (26), pp. 1625

- 1638, 1994.
Trout Unlimited. 2008. An Economic Benefit Analysis for Abandoned Mine Drainage Remediation in the West Branch Susquehanna River Watershed, Pennsylvania. Prepared by Downstream Strat egies.
Viscusi, Kip W, Joel Huber, and Jason Bell. 2008. The Economic Value of Water Quality. Environmental
and Resource Economics 41:169

- 187Vollenweider, R.A. 1976. Advances in defining critical loading levels for phosphorus lake eutrophication. As cite d in Schwarz et al. 2006. Memorie dell’Instituto Italiano di Idrobiologia 33:53

- 83.
Welle, P. G. and Hodgson (2011). "Property Owners’ Willingness To Pay For Restoring Impaired Lakes:
A Survey In Two Watersheds of the Upper Mississippi River Basin." Journa l of Applied Business and Economics 12(1): 81

- 94.
Welle, Patrick G. 1986. Potential Economic Impacts of Acid Deposition: A Contingent Valuation Study of Minnesota. Dissertation, University of Wisconsin Madison.
Wey, K. A. 1990. Social Welfare Analysis of Congestion and Water Quality of Great Salt Pond, Block Island, Rhode Island. Dissertation, University of Rhode Island.
Whitehead, John C., and P.A. Groothuis. 1992. Economic Benefits of Improved Water Quality: a case study of North Carolina's Tar Pam lico River. Rivers. 3: 170 178.
Whitehead, John C., G.C. Blomquist, T.J. Hoban, W.B. Clifford. 1995. Assessing the Validity and Reliability of Contingent Values: A Comparison of On

- Site Users, Off

- Site Users, and Non

- users. Journal of Environmental Ec onomics and Management. Vol. 29.
Whitehead, John C., Hoban, T., and Clifford, W. 2002. Landowners' Willingness to Pay for Water Quality Improvements: Jointly Estimating Contingent Valuation and Behavior with Limited Information. White paper developed in part by US EPA, NCDENR and the College of Agriculture and Life Sciences at
NSCU.
Whittington, Dale, G. Cassidy, D. Amaral, E. McClelland, H. Wang and C. Poulos. 1994. The Economic Value of Improving the Environmental Quality of Galveston Bay. Department of Environmental Sciences and Engineering, University of North Carolina at Chapel Hill.



### C.5 : Studies Used in Open Space Valuation Meta

- Analysis
Abt Associates developed a meta

- analysis of open space benefits to residential property owners to support

benefit s analysis for the proposed post

- construction rulemaking. Abt Associates will update this PQAPP with the results of ongoing efforts to update the meta

- analysis. Current metadata include:

Abbott, J. K., & Kl aiber, H. A. 2010. Is all space created equal? Un covering the relationship between competing land uses in subdivisions. Ecological Economics, 70 (2), 296

- 307. doi: 10.1016/j.ecolecon.2010.09.001

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 84 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Acharya, G., & Bennett, L. L. 2001. Valuing Open Space and Land

- Use Patterns in Urban Watersheds.

The Journal o f Real Estate Finance and Economics, 22 (2

- 3), 221

- 237.
Anderson, S. T., & West, S. E. 2006. Open space, residential property values, and spatial context. [Article]. Regional Science and Urban Economics, 36 , 773

- 789.
Bark, R. H., Osgood, D. E., Colby, B. G., Katz, G., & Stromberg, J. 2009. Habitat preservation and restoration: Do homebuyers have preferences for quality habitat? [Article]. Ecological Economics, 68

, 1465

- 1475.
Bin, O. 2005. A semiparametric hedonic model for valuing wetlands. Applied Economics Letters, 12 (10),
597

- 601. doi: 10.1080/13504850500188505
Bin, O., & Polasky, S. 2005. Evidence on the Amenity Value of Wetlands in a Rural Setting. Journal of Agricultural and Applied Economics, 37 (3), 589

- 602.

Bolitzer, B., & Netusil, N. R. 200 0. The impact of open spaces on property values in Portland, Oregon. Journal of Environmental Management, 59 , 185

- 193.
Bowman, T., Thompson, J., & Colletti, J. 2009. Valuation of open space and conservation features in
residential subdivisions. Journal of Environmental Management, 90 , 321

- 330.

Cho, S.

- H., Roberts, R. K., & Kim, S. 2011

. Negative externalities on property values resulting from water impairment: the case of Pigeon River watershed . Paper presented at the Agricultural & Applied

Economics Asso ciation’s 2011 AAEA & NAREA Joint Annual Meeting, Pittsburgh.
Cho, S. H., Bowker, J. M., & Park, W. M. 2006. Measuring the Contribution of Water and Green Space Amenities to Housing Values: An Application and Comparison of Spatially

- weighted Hedonic Model s. In
S.

- H. Cho, J. M. Bowker & W. M. Park (Eds.).

Cho, S. H., Clark, C . D., Park, W. M., & Kim, S. G. 2009. Spatial and Temporal Variation in the Housing Market Values of Lot Size and Open Space. [Article]. Land Economics, 85 , 51

- 73.
Cho, S. H., Poudyal

, N. C., & Roberts, R. K. 2008 . Spatial analysis of the amenity value of green open

space. Ecological Economics, 66 , 403

- 416.

Donov an, G. H., & Butry, D. T. 2010 . Trees in the city: Valuing street trees in Portland, Oregon. Landscape and Urban Planning, 94 (2), 77

- 83. doi: DOI: 10.1016/j.landurbplan.2009.07.019

Do ss, C. R., & Taff, S. J. 1996. The Influence of wetland type and wetland proximity on residential property values. Journal of Agricultural and Resource Economics, 21 (1), 120

- 129.
Geoghegan, J., Wai n ger, L. A., & Bockstael, N. E. 1997. Spatial landscape indices in a hedonic framework: an ecological economics analysis using GIS. Ecological Economics, 23 , 251

- 264.

Hardie, I., Licht enberg, E., & Nickerson, C. J. 2007. Regulation, open space, and the va lue of land undergoing residential subdivision. Land Economics, 83 , 458

- 474.

Heintzelman, M. A. 2010 . Measuring the property

- value effects of local land use and preservation
referenda. Land Economics, 86 , 22

- 47.

Irwin, E. G. 2002 . The effects of open spa ce on residential property values. Land Economics, 78 , 465

- 480.
Kaufman, D., & Cloutier, N. 2006. The Impact of Small Brownfields and Greenspaces on Residential Property Values. Journal of Real Estate Finance & Economics, 33 (1), 19

- 30. doi: 10.1007/s11146

- 006

- 8272

- 7

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 85 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Kopits, E., McConnell, V., & Walls, M. 2007. The trade

- off between private lots and public open space in
subdivisions at the urban

- rural fringe. [Proceedings Paper]. American Journal of Agricultural Economics,
89 , 1191

- 1197.

Lutzenhi ser, M., & Netusil, N. R. 2001 . The effect of open spaces on a home's sale price. Contemporary Economic Policy, 19 (3), 291

- 298.
Mahan, B. L., Polasky, S., & Adams, R. M. 2000. Valuing Urban Wetlands: A Property Price Approach. Land Economics, 76 (1), 100

- 113.

Munro e, D. K. 2007 . Exploring the determinants of spatial pattern in residential land markets: amenities and disamenities in Charlotte, NC, USA. Environment and Planning B

- Planning & Design, 34 , 336

- 354.

Netusil, N. R. 2006. Economic Valuation of Riparian Corr idors and Upland Wildlife Habitat in an Urban

Watershed. Journal of Contemporary Water Research & Education (134), 39

- 45.
Netusil, N. R., Chattopadhyay, S., & Kovacs, K. F. 2010. Estimating the demand for tree canopy: a

second

- stage hedonic price analysis in Portland, Oregon. Land Economics, 86 (2), 281

- 293.
Ready, R., & Abdalla, C. 2003 . GIS analysis of land use on the rural

- urban fringe: The impact of land use and potential local disamenities on residential property values and on the location of residenti al development in Berks County, Pennsylvania Staff Paper 364 (pp. 60). University Park: Pennsylvania State University, Department of Agricultural Economics and Rural Sociology.
Sander, H., Polasky, S., & Haight, R. G. 2010. The value of urban tree cover: A hedonic property price model in Ramsey and Dakota Counties, Minnesota, USA. Ecological Economics, 69 (8), 1646

- 1656. doi: DOI: 10.1016/j.ecolecon.2010.03.011

San der, H. A., & Polasky, S. 2009 . The value of views and open space: Estimates from a hedonic pri cing model for Ramsey County, Minnesota, USA. Land Use Policy, 26 , 837

- 845.
Saphores, J.

- D., & Li, W. 2012. Estimating the value of urban green areas: A hedonic pricing analysis of the single family housing market in Los Angeles, CA. Landscape and Urban P lanning, 104 (3

– 4), 373

- 387. doi: 10.1016/j.landurbplan.2011.11.012
Shultz, S. D., & King, D. A. 2001. The use of census data for hedonic price estimates of open

- space amenities and land use. Journal of Real Estate Finance and Economics, 22 (2), 239

- 239.

Sm ith, V. K., Poulos, C., & Kim, H. 2002. Treating open space as an urban amenity. Resource and Energy Economics, 24 , 107

- 129.

Stetler, K. M., Venn, T. J., & Calkin, D. E. 2010. The effects of wildfire and environmental amenities on property values in north west Montana, USA. Ecological Economics, 69 (11), 2233

- 2243. doi: DOI: 10.1016/j.ecolecon.2010.06.009
Thorsnes, P. 2002. The value of a suburban forest preserve: estimates from sales of vacant residential building lots. Land Economics, 78 (3), 426

- 441.

Towe, C. 2009 . A valuation of subdivision open space by type. [Proceedings Paper]. American Journal of Agricultural Economics, 91 (5), 1319

- 1325.
Troy, A., & Grove, J. M. 2008. Property values, parks, and crime: A hedonic analysis in Baltimore, MD.

Landsca pe and Urban Planning, 87 , 233

- 245.
White, E. M., & Leefers, L. A. 2007. Influence of natural amenities on residential property values in a rural setting. Society and Natural Resources, 20 , 659

- 667.

### Appendix C

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 86 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. Williams, E. S., & Wise, W. R. 2009. Economic impacts o f alternative approaches to storm

- water management and land development. Journal of Water Resources Planning and Management

- Asce, 135

, 537

- 546.

### Appendix D

Abt Associates PQAPP Contract EP

- C

- 13

- 039 July 2014 ▌ 87 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal. 

### Appendix D : Telephone Contact Record

Telephone Contact Report

Work Assignment and Task: 

### [Title and number]

Call Date: 

### [Date and Time]

Attendees:

[Full Name, Affiliation]

Topic Discussed/Question Asked (1)



###  [Information obtained/Answer to question]

Topic Discussed/Question Asked (2)



###  [Information obtained/Answer to question]

Topic Discussed/Question Asked (3)



###  [Information obtained/Answer to question]