Programmatic Quality Assurance Project 

Plan (PQAPP) for Economic, 

Environmental, and Regulatory Analytical 

and Evaluation Support for Clean Water 

Regulations 

(Contract EP-C-13-039)

Submitted to:

U.S. EPA/OW/OST/EAD (4303T)

EPA West, Room 6233D

1200 Pennsylvania Ave., N.W.

Washington, DC 20460

Attn: Marion Kelly

Kelly.Marion@epa.gov

Submitted by:

Abt Associates

55 Wheeler Street

Cambridge, MA 02138

Contract No. EP-C-13-039

July 31, 2014
Abt Associates PQAPP EP-C-13-039 July 2014 i

Revision History

Revision 

Number

Date 

Submitted Date Approved Record of Activity and Scope of Revisions

0 7/31/14 8/7/14 Original PQAPP
Abt Associates PQAPP EP-C-13-039 July 2014 ii

Title and Approval Sheet

Project Title: Programmatic Quality Assurance Project Plan (PQAPP) for Economic, 

Environmental, and Regulatory Analytical and Evaluation Support for Clean 

Water Regulations (Contract EP-C-13-039)

Organization Name: Abt Associates Inc.

Date: July 2014

Revision Number: 0

A signature below indicates each individual’s approval of this Plan, and commitment to follow the 

procedures it describes.

Abt Associates Approval Signatures

7/31/14

Michael Fisher, Project Manager, Contract No. EP-C-13-039 Date

7/30/14

Susan Day, Quality Assurance Manager, Contract No. EP-C-13-039 Date

7/30/14

Eloise Castillo, Deputy Work Assignment Manager, No. 0-01 Date

7/30/14

Anna Belova, Project Quality Assurance Advisor Date

7/30/14

Isabelle Morin, Project Quality Assurance Advisor Date

7/30/14

Viktoria Zoltay, Project Quality Assurance Advisor Date
Environmental Protection Agency Approval Signatures 

Marion Kelly, Work Assignme Manager, Engineering and Analysis Division 

Quality Assurance Officer 

t() ~ ~~ ~ Ahmar Siddiqui, Project o lffCer, Contract No. EP-C-13-039 

Julie J;lewitt, Economic arid· Environmental Assessment Branch Chief 

Ab------------------~------- t Associates PQAPP EP-C-13-039 

v'1: ( o·r/ ?»'If 

Date 

July 2014 iii 
Abt Associates PQAPP EP-C-13-039 July 2014 iv

Distribution List

EPA and Abt Associates will distribute the approved PQAPP and any subsequent revisions to the 

following individuals:

U.S. Environmental Protection Agency [Engineering Analysis Division, Office of Science and 

Technology, 1200 Pennsylvania Avenue, NW (4303T), Washington, DC 20460]:

 Marion Kelly, EAD Quality Assurance Coordinator, (202) 566-1045, Kelly.Marion@epa.gov

 Ahmar Siddiqui, Project Officer, Contract No. EP-C-13-039; (202) 566-1044, 

Siddiqui.Ahmar@epa.gov

 Julie Hewitt, Economic and Environmental Assessment Branch Chief, (202) 566-1031,

Hewitt.Julie@epa.gov

 Work Assignment Manager, specified in Work Assignment

Abt Associates Inc. (Offices at: 55 Wheeler Street, Cambridge, MA 02138-1168; and 4550 Montgomery 

Avenue, Suite 800 North, Bethesda, MD 20814):

 Michael Fisher, EPA Contract EP-C-13-039 Project Manager, (617) 349-2526, 

Michael_Fisher@abtassoc.com

 Susan Day, EPA Contract EP-C-13-039 Quality Advisor, (617) 349-2768, 

Susan_Day@abtassoc.com

 Elena Besedin, Senior Benefits Economist Key Personnel, (617) 349-2770, 

Besedin_Elena@abtassoc.com

 Isabelle Morin, Senior Environmental Assessor Key Personnel, (617) 349-2667, 

Isabelle_Morin@abtassoc.com

 Eloise Castillo, Senior Cost Economist Key Personnel, (301) 347-5701, 

Eloise_Castillo@abtassoc.com

 Anna Belova, Project Quality Assurance Advisor, (301) 347-5304, Anna_Belova@abtassoc.com

 Victoria Zoltay, Project Quality Assurance Advisor, (617) 520-2721, 

Viktoria_Zoltay@abtassoc.com 

 Dan Basoli, Work Assignment Manager, (617) 349-2344, Dan_Basoli@abtassoc.com

 Debra Fleischer, Work Assignment Manager, (301) 347-5396, Debra_Fleischer@abtassoc.com

 Emily Giovanni, Work Assignment Manager, (301) 347-5398, Emily_Giovanni@abtassoc.com

 Tatiana Melnikova, Work Assignment Manager, (617) 520-2824, 

Tatiana_Melnikova@abtassoc.com

 James Palardy, Work Assignment Manager, (301) 347-5235, James_Palardy@abtassoc.com

 Ryan Stapler, Work Assignment Manager, (617) 520-3524, Ryan_Stapler@abtassoc.com
Abt Associates PQAPP EP-C-13-039 July 2014 v

Programmatic Quality Assurance Project Plan Content 

Exhibit 1 provides a crosswalk of the Programmatic Quality Assurance Project Plan to elements 

identified in EPA Requirements for Quality Management Plans (QA/R-2), March 2001 (Reissued 

May 2006).

Exhibit 1: Crosswalk of QAPP elements to PQAPP section.

Element Page/Section

Group A. Project management

A1. Title and Approval Sheet Page ii

A2. Table of Contents Page vii

A3. Distribution List Page iv

A4. Project/Task Organization Section 2.1

A5. Problem Definition and Background Section 2.2

A6. Project/Task Description Section 2.3

A7. Overall Quality Objectives and Criteria Section 2.4

A8. Special Training/Certifications Section 2.5

A9. Project-Level Documentation and Records Section 2.6

Group B. Data Generation and Acquisition

Existing Data Gathering Project

B1. Data Needs Section 3.1.1

B2. Potential Data Sources Section 3.1.2

B3. Criteria for Selecting Data Sources Section 3.1.3

B4. Data Value Selection Approach Section 3.1.4

B5. Resolving Data Gaps Section 3.1.5

B6. Data Gathering Documentation and Records Section 3.1.6

Primary Data Gathering Project

B1. Sampling Process (Experimental) Design Section 4.1.1

B2. Sampling Methods Section 4.1.2 (Data Gathering Methods and Steps)

B3. Sample Handling and Custody Section 4.1.3 (Completed Questionnaire Handling and 

Custody)

B4. Analytical Methods Not applicable1

B5. Quality Control Not applicable1

B6. Instrument/Equipment Testing, Inspection, and 

Maintenance

Not applicable1

B7. Instrument/Equipment Calibration and 

Frequency

Not applicable1

B8. Inspection/Acceptance of Supplies and 

Consumables

Not applicable1

B9. Non-direct Measurements Section 3.1.2

B10. Data Management Section 4.1.4

Group C. Data Management

Existing Data Gathering Project
Abt Associates PQAPP EP-C-13-039 July 2014 vi

C1. Standardization of Data Elements Section 3.2.1

C2. Data Entry Section 3.2.2

C3. Merging or Uploading Electronic Data from 

Existing Sources

Section 3.2.3

C4. Data Review Section 3.2.4

C5. Data Storage and Manipulation Section 3.2.5

Primary Data Gathering Project

C1. Assessments and Response Actions Section 4.1.5

C2. Reports to Management Section 4.1.6

Group D. Data Quality Assessment and Use

Existing Data Gathering Project

D1. Data Quality Verification and Data Quality 

Reporting

Section 3.3.1

D2. Use/Analysis of the Existing Data Section 3.3.2

D3. Methodology Documentation and Conceptual 

Review

Section 3.3.3

D4. Technical Review of the Data Analysis Section 3.3.4

D5. Final Verification of Data Analysis and 

Reconciliation with User Requirements

Section 3.3.5

Primary Data Gathering Project

D1. Data Review, Verification, and Validation Section 4.2.1

D2. Verification and Validation Methods Section 4.2.1

D3. Reconciliation with User Requirements Section 4.2.2

1. Project does not involve laboratory or field measurements.
Abt Associates PQAPP EP-C-13-039 July 2014 vii

Contents

1. Introduction ...............................................................................................................................3

2. Project Management .................................................................................................................5

2.1 Project Organization..........................................................................................................5

2.2 Problem Definition/Background........................................................................................9

2.3 Project/Task Description ................................................................................................. 14

2.4 Overall Data Quality Objectives and Criteria .................................................................. 16

2.4.1 Environmental Assessments ............................................................................... 16

2.4.2 Economic/Financial Analyses............................................................................. 17

2.4.3 Benefits Valuation .............................................................................................. 19

2.5 Training and Certification................................................................................................ 22

2.6 Project-Level Documents and Records............................................................................ 23

2.6.1 CBI..................................................................................................................... 24

2.6.2 Deliverables........................................................................................................ 24

2.6.3 Quality Audits .................................................................................................... 25

3. Existing Data............................................................................................................................27

3.1 Data Gathering ................................................................................................................ 27

3.1.1 Data Needs ......................................................................................................... 27

3.1.2 Potential Data Sources........................................................................................ 40

3.1.3 Criteria for Selecting Data Sources..................................................................... 45

3.1.4 Data Value Selection Approach.......................................................................... 48

3.1.5 Resolving Data Gaps .......................................................................................... 49

3.1.6 Data Gathering Documentation and Records...................................................... 49

3.2 Data Management............................................................................................................ 51

3.2.1 Standardization of Data Elements....................................................................... 51

3.2.2 Data Entry........................................................................................................... 52

3.2.3 Merging or Uploading Electronic Data from Existing Sources........................... 52

3.2.4 Data Review ....................................................................................................... 53

3.2.5 Data Storage and Manipulation .......................................................................... 53

3.3 Data Quality Assessment and Use ................................................................................... 54

3.3.1 Data Quality Verification and Data Quality Reporting ....................................... 54

3.3.2 Use/Analysis of Existing Data/Data Operations ................................................. 54

3.3.3 Methodology Documentation and Conceptual Review....................................... 56

3.3.4 Technical Review of the Data Analysis.............................................................. 57

3.3.5 Final Verification of Data Analysis and Reconciliation with User Requirements57

4. Primary Data............................................................................................................................59

4.1 Data Gathering ................................................................................................................ 60

4.1.1 Sampling Process (Experimental) Design........................................................... 60

4.1.2 Data Gathering Methods and Steps..................................................................... 61

4.1.3 Completed Questionnaire Handling and Custody ............................................... 62

4.1.4 Data Management............................................................................................... 63

4.1.5 Assessment and Response Actions ..................................................................... 64

4.1.6 Reports to Management...................................................................................... 64

4.2 Data Quality Assessment and Use ................................................................................... 64

4.2.1 Data Review, Verification, and Validation ......................................................... 64

4.2.2 Reconciliation with User Requirements.............................................................. 65

4.3 Telephone Interviews ...................................................................................................... 66
Abt Associates PQAPP EP-C-13-039 July 2014 viii

5. References...............................................................................................................................67
Abt Associates PQAPP EP-C-13-039 July 2014 i

Abbreviations

BFT Benefit Function Transfer

BMP Best Management Practice

BUVD Benefit Use Valuation Database

CATI Computer Assisted Telephone Interviewing

CBI Confidential Business Information

CWA Clean Water Act

EAD Engineering and Analysis Division

EIA Energy Information Administration

ELGs Effluent Limitations Guidelines

EPA Environmental Protection Agency

ESD Environmental site design

EVRI Environmental Valuation Resource Inventory

FAQ Frequently Asked Questions

HUC Hydrologic Unit Code

ICIS Integrated Compliance Information System

ICR Information collection request

IRIS Integrated Risk Information System

IT Information Technology

LID Low impact development

NAICS North American Industry Classification System

NPDES National Pollutant Discharge Elimination System

OST Office of Science and Technology

OW Office of Water

PDF Portable document format

PQA Project Quality Assurance

PQAPP Programmatic Quality Assurance Project Plan

PWS Performance Work Statement

QA Quality assurance

QA/QC Quality assurance and quality control

QAC Quality Assurance Coordinator

QAPP Quality Assurance Project Plan

QMP Quality Management Plan

RFA Regulatory Flexibility Act

SEC Securities and Exchange Commission

SQAPP Supplemental Quality Assurance Project Plan

SWAT Soil and Water Assessment Tool
Abt Associates PQAPP EP-C-13-039 July 2014 ii

UMRA Unfunded Mandates Reform Act 

USGS United States Geological Survey

WAM Work Assignment Manager

WQI Water quality index

WQL Water quality ladder

WSUT Weak structural utility theoretic

WTP Willingness to pay
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 3

1. Introduction

This Programmatic Quality Assurance Project Plan (PQAPP) covers quality assurance and quality control 

(QA/QC) applicable to the collection, generation, evaluation, analysis, and use of environmental data by 

Abt Associates Inc. to provide Economic, Environmental, and Regulatory Analytical and Evaluation 

Support for Clean Water Regulations (Contract EP-C-13-039).

The Office of Water (OW) of the United States Environmental Protection Agency (EPA, or the Agency)

is responsible for implementing the Clean Water Act (CWA). The Engineering and Analysis Division 

(EAD) within OW, and the Office of Science and Technology (OST), develops national industrial 

discharge regulations known as effluent guidelines. The primary purpose of the Performance Work 

Statement (PWS) for Contract EP-C-13-039 is to support economic and environmental analyses related to 

the development of effluent guidelines regulations. This support encompasses:

 Environmental assessment

 Economic and financial analysis

 Environmental benefits valuation.

Other divisions within the OW that use this contract to support their programmatic activities may need to 

develop a project-specific Quality Assurance Project Plan (QAPP).

EPA requires, through its quality assurance (QA) policy, that a QAPP be approved prior to commencing 

work that involves the generation, collection, analysis, and use of environmental data. Under Contract EPC-13-039, Abt Associates Inc. (Abt Associates) will perform work that involves these types of

environmental data operations (see Table 1-1). For example, support to EPA in analyzing the economic 

impacts of regulatory options may entail a wide range of data-related activities. These activities may 

include generation of primary data by surveying affected entities, collection of existing data from public 

sources, evaluation and assessment of data to ensure completeness and representativeness, analysis of the 

impacts of increased costs on various entities, and use of the data to support EPA in making decisions 

related to implementing the CWA, Executive Orders, or other requirements. To facilitate compliance with 

the policy, and to assist EAD in ensuring the quality of the work products, Abt Associates has prepared 

this PQAPP for work involving both primary and existing environmental data. Primary data are data 

generated by EAD or under its direction to support the decision at hand; existing data are any data not

generated directly by EAD to support the decision at hand. 

Table 1-1: Examples of work that involve the collection, generation, evaluation, analysis, or 

use of environmental data.

Item Examples

Data generation Field studies, laboratory studies, and modeling output generation 

Data collection Field surveys, questionnaires, literature searches, and third party data collection

Data evaluation Data inspection, review, assessment, and validation evaluation

Data analysis Statistical, engineering, and economic analysis, and testing, evaluation, and validation of

analysis methods and models; database creation, data extraction, and data manipulation

Data use Support for EPA decisions, regulations, policy, publications, or tools (including effluent

guidelines, 304(m) program, standards, environmental assessments, and models, tools, 

or reports disseminated by EPA to assist other organizations in implementing 

environmental programs)

EAD develops effluent guidelines in a progressively elaborative manner as it gathers and evaluates data, 

and evaluates new needs and gaps. Thus, the activities we describe in this document, based on past 

experience and current work assignments, may not be comprehensive of all work undertaken under this 

contract. As needed and directed by EPA, Abt Associates will prepare supplemental QAPPs (SQAPPs) or 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 4

traditional QAPPs for individual work assignments. SQAPPs supplement the approved PQAPP by 

providing the additional detail needed to describe the specific goals and objectives of the project or tasks 

supported under the work assignment, as well as the specific QA strategies and technical activities that 

the project team will use to support the objectives of the work assignment project or task. Traditional 

QAPPs for individual work assignments are stand-alone documents that focus on the goals, objectives, 

QA strategies, and technical activities needed to support a single work assignment or project. Abt 

Associates will then incorporate changes into this PQAPP as appropriate.

Abt Associates developed this PQAPP in accordance with EPA’s Guidance for Quality Assurance Project 

Plans (EPA/240/R-02/009), and the suite of alternative elements that EAD designed to improve the 

quality of QAPPs addressing existing data (EPA EAD QAPP Guidance). All project deliverables that Abt 

Associates prepares under this contract will meet EPA’s standards of transparency, reproducibility, 

objectivity, integrity, and utility as specified in EPA’s Guidelines for Insuring and Maximizing the 

Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection 

Agency (EPA/260R-02-008, October 2002), referred to as “EPA’s Information Quality Guidelines.”

The remainder of this document is organized as follows:

 Section 2 describes Project Management QAPP elements (Elements A in EPA guidance).

 Section 3 describes QAPP elements pertinent to existing data collection, management, 

assessment, and use.

 Section 4 describes QAPP elements pertinent to primary data collection, management, 

assessment, and use.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 5

2. Project Management

This section addresses project management elements, including project organization, roles and 

responsibilities, and goals and objectives. In addition, this section presents Abt Associates’ approach for 

ensuring that all members of the project team understand the goals and procedures applicable to this 

program. Abt Associates meets quality objectives through setting clear project organization and 

objectives; incorporating quality assurance within all aspects of collecting, managing, and using data;

assigning appropriate staff with the requisite expertise and training; ongoing verification and review of 

work products and QA procedures implementation; and documentation.

2.1 Project Organization

EAD will oversee the program, and individual projects, covered by this PQAPP. Figure 2-1 shows the 

organization and roles within EAD.

Figure 2-1: Project Organizational Chart.

The Economic and Environmental Assessment Branch Chief, Ms. Julie Hewitt, will be responsible for

oversight of project activities on behalf of EPA. In this capacity, Ms. Hewitt will be responsible for:

 Participating in project planning activities and providing direction regarding project scope, 

resources, design and quality requirements

 Approving the final project design and quality objectives

 Assigning project staff within the EAD Economic and Environmental Assessment Branch, and 

securing all staff and financial resources necessary to design and implement the project

 Assisting and providing technical input to EPA Work Assignment Manager (WAM) as needed
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 6

 Approving this PQAPP, any EAD SQAPPs, and other EAD quality planning documents 

including project-specific QAPPs 

 Overseeing project budgets and schedules.

The EPA WAM for a specific assignment will be responsible for day-to-day management and direction of 

all study activities. In this capacity, the EPA WAM will be responsible for:

 Managing the work assignment under which Contractor support will be provided, directing all 

Contractor activities associated with implementation of the project, and ensuring that the 

Contractor is provided with the resources necessary to implement the project

 Producing (or directing production of) a written project plan that details project objectives, tasks, 

schedules, roles, and responsibilities

 Managing project budgets and schedules, and notifying the Branch Chief of problems or delays 

that may impact project objectives, budgets, or schedules

 Coordinating with the EAD Quality Assurance Coordinator (QAC) to define and document 

quality objectives and QA/QC strategies for the project and to produce this PQAPP and any 

SQAPPs, as needed

 Adhering to all applicable EPA requirements concerning the quality of information produced 

under this project (e.g., OW’s Quality Management Plans, EPA’s Information Quality 

Guidelines)

 Ensuring that all staff involved with the project (inside and outside of EAD) adhere to the 

requirements of this PQAPP.

The EAD QAC, Marion Kelly, will be responsible for:

 Providing guidance to the EPA WAM concerning the Agency’s quality requirements and 

recommending QA/QC strategies

 Assisting the EPA WAM as needed with implementation of QA/QC strategies

 Reviewing and approving this PQAPP, any SQAPPs, and other quality planning documents 

including project-specific QAPPs.

The EAD QAC reports to the EAD Director, and is independent from the technical managers who oversee 

the program and projects.

Figure 2-1 also shows Abt Associates’ organization as pertains to QA/QC activity for this contract. As 

directed by the EPA WAM, Abt Associates and its subcontractors and consultants will be responsible for 

drafting, revising, and/or finalizing study materials. Examples of these materials include this PQAPP, any 

SQAPPs needed, surveys or questionnaires, analytic files, memoranda, and a final study report that 

includes a distinct section addressing QA/QC (see Section 3.3.5). Abt Associates will provide scientific 

and statistical support to EPA in resolving problems and evaluating data. As shown in the figure, we have 

organized our support to ensure the independence of the Abt Associates Quality Assurance Manager from 

the Project Manager.

Table 2-1 provides a summary of the roles and QA responsibilities of Abt Associates staff, with detailed 

descriptions following the table.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 7

Table 2-1: QA Roles and Responsibilities.

Position Quality-Related Responsibility

Project Manager (Michael Fisher)  Provide managerial oversight of the contract

 Work with the Abt Associates WAM to assign a PQA Advisor

 Review and approve PQAPP, SQAPPs, and other QA planning documents 

such as a project-specific QAPP

Quality Assurance Manager

(Susan Day)

 Implement, monitor, and maintain the procedures described in the Quality 

Management Plan, PQAPP, SQAPPs, and other QA planning documents 

such as a project-specific QAPP

 Ensure qualified Project Quality Assurance Advisors assigned to every WA

Project Quality Assurance 

Advisor (TBD on project-specific 

basis)

 Ensure and enhance project quality

 Provide input for SQAPP or other QA planning document such as a 

project-specific QAPP (if needed), and approve

 Technical review

Work Assignment Manager

(TBD on project-specific basis)

 Develop and approve SQAPP or other QA planning document such as a 

project-specific QAPP (if needed)

 Work with the Project Manager to identify a Project Quality Assurance 

Advisor

 Organize the work and apply resources to accomplish the objectives

 Monitor work, schedules, and budgets

 Ensure sufficient time and resources are allocated to QA/QC procedures

 Ensure project team members, including subcontractors and consultants, 

are aware of and implement PQAPP and SQAPP procedures

 Report on QA/QC activities

Modeler, Spreadsheet or 

Database Developer (TBD)

 Review and adhere to PQAPP, and SQAPP or other QA planning 

document such as a project-specific QAPP (if needed)

 Develop model, spreadsheet, or database development following best 

practices

 Participate in routine and ad-hoc conference calls to discuss project status, 

issues, and questions

 Notify Abt Associates Work Assignment Manager of issues that will affect 

project budgets, schedules, or goals

Additional Project Team 

Members (TBD on project-specific 

basis)

 Review and adhere to PQAPP, and SQAPP or other QA planning 

document such as a project-specific QAPP (if needed)

 Participate in routine and ad-hoc conference calls to discuss project status, 

issues, and questions

 Notify Abt Associates Work Assignment Manager of issues that will affect 

project budgets, schedules, or goals

PQA = Project Quality Assurance

PQAPP = Programmatic Quality Assurance Project Plan

QA = quality assurance

QA/QC = quality assurance/quality control

SQAPP = Supplemental Quality Assurance Project Plan

TBD = to be determined

WAM = work assignment manager

Mr. Michael Fisher, the Abt Associates Project Manager, will manage the activities by Abt Associates 

and its subcontractors and consultants. Mr. Fisher reviews and approves this PQAPP, and any SQAPPs 

prepared for individual WAs. He also assigns the Abt Associates WAM, and works with the WAM to 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 8

designate the assignment-level Project Quality Assurance (PQA) Advisor. If any QA concerns arise in the 

course of the work that cannot be resolved within the project team, Mr. Fisher will refer these concerns to 

the Contract-Level Quality Assurance Manager, Ms. Susan Day.

Ms. Susan Day maintains overall responsibility for implementing, monitoring, and maintaining the 

procedures described in the Abt Quality Management Plan (QMP) as they apply to this contract 

(including ensuring that the Project Manager and WAMs have assigned a qualified PQA Advisor for each 

work assignment), reviews and approves the PQAPP and any SQAPPs, and verifies that WAMs and PQA

Advisors adhere to procedures outlined therein. Ms. Day is organizationally separate from Mr. Fisher, the 

WAMs, and the day-to-day work on the projects, reporting directly to Andrew Stoeckle, Vice 

President/Principal Associate.

The Abt Associates WAM works with the Project Manager to identify and assign a technically qualified 

individual as the PQA Advisor. The Abt Associates WAM also typically develops the SQAPP, if needed. 

Depending on the nature of the work assignment, the WAM may work with a subcontractor or consultant 

to develop the SQAPP, or parts thereof. The WAM ensures that all staff working on the individual 

projects are aware of the PQAPP, and have received a copy of the document.

The Abt Associates WAM writes, or directs the writing of a PQA Plan. The PQA Plan describes the key 

steps in the project, the level of PQA Advisor involvement, and the project technical review requirements. 

Susan Day, the Quality Assurance Manager for this contract, and Abt Associates’ division management 

must approve the plan; the WAM is responsible for allocating appropriate time for review of work 

products into the project or task schedule, and for dedicating adequate resources from the work 

assignment budget for technical review and QA oversight. 

At the start of each project, and at various times during project execution, the Abt Associates WAM

reviews the QA procedures with the task leads to confirm that the leads are implementing the procedures, 

and discuss any needed improvements. To the extent that such reviews identify needed changes in 

procedures for conducting analyses or conducting QA/QC, we will discuss these changes with the EPA 

WAM. We note these discussions and changes in monthly project status reports to the EPA WAM, and 

may incorporate the revised procedures in future versions of this PQAPP.

The Abt Associates PQA Advisor ensures and enhances project quality by providing project staff access 

to the experience and judgment of a senior staff member who is not immediately involved in the project 

work, and by providing ongoing independent technical review of project work. 

The PQA Advisor participates in quality enhancement discussions with the Abt Associates WAM or other 

project staff at several key points during the project (e.g., see Section 2.6.3 on quality reviews). These 

discussions provide the project staff with the benefits of an objective assessment of plans, progress to 

date, and potential pitfalls. The extent and nature of such discussions depend on the complexity of the 

project, the potential for quality concerns, and the cost associated with waiting to address problems until 

the technical review. 

The PQA Advisor also provides technical review of project work. The technical review requirement is 

essentially a peer review requirement. This requirement is consistent with the nature of our work, which is 

innovative and project-specific rather than routine, and often involves considerable expertise and 

judgment in applying that expertise. In general, the Abt Associates WAM may not submit any substantive 

project deliverable or product to a client in either draft or final form without the technical reviewer’s 

certification that it is technically acceptable in terms of content, methods, and exposition. For example, in 

reviewing a report, the reviewer usually reviews the draft, marks it up, and discusses problems and 

comments with the authors. Reviewers are concerned not only with methods and interpretation, but also 

with the clarity of the presentation and the application of the findings to the client’s problem. The 

reviewer rereads all or part of the revised draft as necessary, and must approve the draft before we send it 

to the client. (If there are any unresolved quality issues following this review, the PQA Advisor consults 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 9

with the Quality Assurance Manager for the contract, Susan Day, who may involve Division management 

in resolving the issues.) The project schedule must provide time for both review and response to reviewer 

comments.

Technical review for projects may also include after-action reviews. These reviews enhance future 

performance on similar projects through review of the challenges and accomplishments of completed 

projects. As described above, to the extent that such reviews identify needed changes in procedures for 

conducting analyses or conducting QA/QC, we will discuss these changes with the EPA WAM, and may 

incorporate the revised procedures in future versions of this PQAPP.

The technical staff for any particular WA may include economists, environmental scientists, and other 

staff in a range of disciplines and expertise, depending on the work assignment. Staff involved in 

collecting, processing, and analyzing data have primary responsibility in implementing QA procedures 

including ensuring that computer programming and development of models, spreadsheets, and databases 

follow best practices, as described in Section 3.2. Best practices serve to minimize errors and maximize 

transparency in data development and analysis, and facilitate quality review of the work product. Staff in 

these capacities review and adhere to the PQAPP and SQAPP, participate in conference calls to discuss 

the work, and notify the WAM of any issues that may affect the project schedules, budgets, and goals.

Abt Associates subcontractors and consultants review and adhere to this PQAPP, and any SQAPPs and 

traditional QAPPs. These project team members participate in conference calls to discuss the work, and 

notify the WAM of any issues that may affect the project schedules, budgets, and goals.

2.2 Problem Definition/Background

The goal of this program is to conduct environmental and economic analyses that inform EAD’s 

development of Effluent Limitations Guidelines (ELGs). In conducting environmental analyses, Abt 

Associates brings EAD access to cost-efficient modeling and government-produced data relevant to 

characterizing pollutant loadings and reductions. With respect to economic analyses, Abt Associates 

conducts work under this program pursuant to the Agency’s guidelines, Guidelines for Preparing 

Economic Analyses, (EPA 240-R-10-001; December 2010), Final Guidance for EPA Rulewriters: 

Regulatory Flexibility Act as amended by the Small Business Regulatory Enforcement Fairness Act (EPA, 

November 2006), Interim Guidance on Considering Environmental Justice During the Development of an 

Action (EPA, July 2010), and other guidance. Additional considerations for determining data and analytic 

needs include the type of demonstration that EPA uses to support policy decisions, depending on 

regulatory requirements. For example, EAD determines ELGs based on the best available technology that 

is economically achievable. In conducting environmental analyses, Abt Associates brings EAD access to 

cost-efficient modeling and government-produced data relevant to characterizing the economic, 

environmental, and financial impacts associated with pollutant loadings and reductions.

The CWA directs EPA to promulgate ELGs that are economically achievable based on available 

technology, by categories or subcategories of industrial point sources. EAD bases these ELGs on specific 

technologies (including process changes) that meet the levels of control prescribed in the CWA. The Act 

prescribes six levels of control based on the type of source (new vs. existing), the type of discharge (direct 

vs. indirect), and the type of pollutants being discharged (conventional, non-conventional, or priority 

pollutants). Chapter 1 of EPA’s Technical Support Document for the Preliminary 2010 Effluent 

Guidelines Program Plan, October 2009, EPA 821-R-09-006 (EPA, October 2009), provides additional 

background information about CWA requirements and the effluent guidelines program.

Developing ELGs is a complex, iterative, multiyear effort that includes several distinct project phases and 

technical disciplines, as shown in Figure 2-2. In developing these ELGs, EAD receives support from

contractors with needed technical expertise and experience. This PQAPP describes the activities that Abt 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 10

Associates, and its subcontractors and consultants on this contract, will perform in working with 

environmental data related to Steps 2, 3, 4, 7, 12, 13c and 13d, and 14 through 17 in Figure 2-2.

Table 2-2 summarizes the questions to be answered by the different types of environmental assessment, 

economic/financial, and benefits analyses, and the work activities that produce the answers. Each WA 

under this contract describes the specific objectives needed to support these goals, including generating, 

collecting, analyzing, and using environmental data to model and assess the environmental impacts of 

pollution controls, estimate the social costs and benefits associated the controls, and compare the impacts, 

costs, and benefits of the alternative regulatory options that EAD may consider. Appendix A provides the 

QA checklist for the activities conducted for each of these analyses.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 11

Figure 2-2: Effluent Guidelines Development Flowchart.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 12

Table 2-2: Goals of the Analyses.

Analyses Work Activities Questions Answered

Environmental 

assessment

 Review and characterize 

the environmental effects 

of pollutant discharges 

 Model the changes in 

environmental conditions 

resulting from regulatory 

options

 What is the impact of pollutant loadings on human health? 

 What is the impact of pollutant loadings on ecosystem services 

provided by water resources?

 Are there ancillary impacts of pollution controls on air and 

terrestrial resources?

 What environmental effects will the options have on affected 

waters and other media?

Cost-benefit 

and economic 

impact analysis

 Estimate private and 

social costs of regulatory 

options and compare to 

social benefits

 Would the options have an annual economic impact of 

$100,000,000 or more?

 Would the options impose incremental costs to state and local 

governments?

 Would the options cause some facilities to encounter severe 

financial stress and close? 

 Would the options cause some facilities moderate financial 

stress (increased challenge and cost in accessing financial 

capital)?

 Which options are economically achievable?

 How many small businesses will incur a significant impact? 

 Would a substantial number or percent of small entities incur a 

significant impact?

 Would consumer prices increase?

 How will the options affect employment?

 Do compliance costs pose barriers to entry for new firms?

 What are the benefits from the options, including primary and 

secondary? 

 How do benefits vary across the options?

 What categories of benefits can be quantified, and monetized?

 What are net benefits across options?

 How are net benefits likely to change with the ability to quantify 

and monetize all benefits, and what is the “break-even” level of 

benefits?

Costeffectiveness 

analysis

 Calculate cost 

effectiveness of 

regulatory options 

 How do reductions in toxic-weighted pollutant loadings to 

surface waters vary with the cost of achieving those reductions 

across the options?

 How do the unit costs of achieving toxic-weighted pollutant 

discharges compare to previously promulgated Effluent 

Limitations Guidelines, considering differences in industrial 

category and available technologies?
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 13

Table 2-2: Goals of the Analyses.

Analyses Work Activities Questions Answered

Environmental 

justice and 

other 

distributional 

analysis

 Analyze the socioeconomic characteristics 

of populations that incur 

the environmental 

damages that would be 

reduced by the 

regulatory options

 Analyze the distribution 

of benefits and price 

increases among 

populations affected by 

the regulatory options

 What populations are affected by pollutant discharges?

 Are minority, low-income, and indigenous populations 

(Environment Justice populations) subject to disproportionate 

environmental and public health impacts?

 How would the options contribute to lessening impacts on these 

populations?

 Would the options have different distributions of benefits across 

the Environmental Justice populations?

 If prices for affected goods and services are likely to increase 

because of the compliance costs, how would these price 

increases and associated increases in consumer costs be 

distributed among various customer groups? 

 Would increases in consumer costs cause undue economic 

burden and/or fall disproportionately on Environment Justice 

populations?

Industry profiles  Compile and describe 

data on facilities and 

entities within the 

industry(ies) affected by 

the regulatory options

 What categories and subcategories of industries discharge the 

pollutants of concern?

 What are baseline economic and financial conditions, pollution 

control technologies, and pollutant loadings?

Industry 

financial survey

development

 Develop questionnaire 

and Information 

Collection Request

 Review and compile 

survey data

 What are the baseline financial circumstances of firms 

employing different pollution control technologies?

Travel cost 

models, 

hedonic implicit 

price models, 

and stated 

preference

survey research

 Develop survey 

instruments

 Compile and analyze 

data to determine 

statistically significant 

factors and relationships

 What is the value of environmental goods and services?

 What is the incremental value of water quality improvements?

Research and 

evaluation of

new methods 

for economic 

analysis related 

to the control of 

water pollution

 Review and compile 

literature, models, and 

existing data

 Are there innovative or streamlined approaches to conducting

cost-benefit analysis, economic impact analysis, costeffectiveness analysis, equity analysis of effluent guidelines 

and standards, or other regulatory and nonregulatory programs 

and initiatives?

Life cycle 

analysis

 Analyze data to identify 

sources of pollutants of 

concern to receiving 

waters

 Are there pollution prevention opportunities?

 What is the merit of regulatory and nonregulatory options 

involving transfer of pollutants from one media to another?

Collection/

preparation of 

reports

 Prepare draft and final 

Regulatory Impact 

Analysis, Benefit-Cost 

Analysis, and 

Environmental 

Assessment and other 

reports

 Do the data, methods, quality assurance/controls, and results 

of studies and analyses support the decisionmaking in the 

regulatory development process?

 Is the decision-making in support of the regulatory development 

process transparent in the record?
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 14

Table 2-2: Goals of the Analyses.

Analyses Work Activities Questions Answered

Review and 

analysis of 

public 

comments

 Compile, parse, and

categorize comments

 Draft responses to 

substantive comments

 Impact of data and analyses provided by public on EPA’s draft 

regulatory selection decisions

Legislative and 

litigation 

support

 Summarize data, 

methods, and key 

findings

 Impact of alternative legislative proposals

 Response to allegations relating to rulemaking or planning 

record

Database 

development 

and 

management

 Assemble information in 

a systematic relational 

framework

 How can databases containing discharge, environmental, and 

economic characteristics of the sectors potentially subject to 

regulation/control be developed and managed to ensure the 

accuracy, reliability, and integrity of the data and information?

Outreach  Draft outreach 

documents such as web 

pages, fact sheets, rule 

summaries, etc.

 Is the regulatory process transparent to stakeholders?

Training and 

logistical 

support

 Organize and facilitate 

meetings and training 

events

 How can stakeholders be trained to implement the rulemaking 

efficiently?

2.3 Project/Task Description

Abt Associates typically conducts analyses under this program starting with an understanding of the 

entities (e.g., plants, sites) and waterbodies that a given policy would affect, and the associated baseline 

conditions relative to the incremental effects of the policy that we evaluate. We then superimpose the 

effects of the policy on baseline conditions to estimate the changes attributable to the policy.

Accordingly, in gathering data, we seek to provide a comprehensive understanding of affected entities 

and/or waterbodies. When the universe of affected entities is well-defined and the number relatively small 

(e.g., a few hundred plants can be readily identified through public sources), we often gather information 

about each individual entity. For example, we may gather data for electricity generating plants from 

public sources such as Department of Energy databases or through surveys of plants to support analysis of 

regulations affecting the electric power industry.

When the universe of affected entities is less well-defined or the number of dischargers is very large (e.g., 

thousands of construction sites or diffuse nonpoint sources), we may instead gather data for a 

representative sample of dischargers or locations that cover a range of operating conditions for the 

industry. For example, analysis of the effects of stormwater management requirements may consider costs 

and environmental effects for a diverse set of development projects and watersheds representing a range 

of site characteristics, geographic locations, existing standards, and compliance approaches. The process 

for identifying a representative sample involves identifying factors that drive the variable being estimated 

(e.g., location), and designing and selecting a sample that includes the different possibilities.

Abt Associates’ analyses for this program typically provide understanding of changes attributable to the 

policy or regulation relative to a baseline representing “business-as-usual” conditions in the absence of 

the policy or regulation. The experimental design therefore also involves defining the appropriate analytic 

baseline considering existing conditions or reasonable projections over the period of analysis, including 

current industry trends, applicable regulations and standards, economic outlook, any applicable climate 

considerations, among others. Data gathering efforts may therefore include, in addition to the information 

above, assembling projections by government and industry trade associations (such as modeled electricity 

demand and supply or projected construction activities), and state and national regulatory requirements.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 15

To support the program goals and achieve the specific project objectives specified in each work 

assignment under this contract, Abt Associates will gather data from a variety of sources (see detailed list

in Table 3-1. The types of data needed for the environmental and economic regulatory analyses include, 

but are not limited to:

 Characteristics of entities that may be affected by the policy or regulation (industry or public 

utilities), including economic/financial data, ownership information, industry sector, number of 

employees, production levels, effluent quantities, constituents

 Characteristics of waterbodies that the policy or regulation may affect, including type, flow, 

condition, water quality, land use within the watershed

 Baseline environmental conditions that may affect policy effectiveness, including land use, point 

and nonpoint sources, existing and projected levels of best management practice or treatment

technology implementation

 Information about water pollution control technology implementation, effectiveness, and costs

 Market conditions and trends in industries potentially affected by policies or regulations (e.g., 

demand, supply, prices, competition)

 Information about resources potentially affected by policies or regulations, including human 

health effects and incidence rates, non-consumptive recreation (e.g., boating, swimming, wildlife 

viewing) activity levels, federally listed threatened and endangered species and location

 Information about value of use and non-use benefits, including existence value of water quality 

improvements, commercial fishery value by species, cost-of-illness

 State and government administrative expenditures

 Data on macroeconomic conditions, wages, and interest rates.

Abt Associates anticipates that the sources of these data will include:

 Public literature and information available via the Internet

 EPA databases that the Agency designed and developed for the particular data need for this 

program

 Other EPA offices that have collected data from the industry and sources of relevance to this 

program

 State, local and other federal agencies

 Other organizations that may be identified during the course of the data identification and 

collection process, such as federal agencies in other countries that are responsible for information 

on cross-boundary pollution. 

Abt Associates anticipates that data collection and identification will be an iterative process in which:

 Initially gathered materials may suggest or point to other sources of existing data

 Initial searches yield excessive amounts of information that, upon further review, we determine to 

be either not directly relevant or of a quality that is less than desired

 Initial searches yield too little data and, therefore, we refine the searches to yield additional 

information

 Initial searches yield data that contains gaps in information needed by EAD.

Abt Associates will evaluate data identified during the searches based on relevance, level of detail, and 

types of information covered, as described in Section 3. After conducting initial searches and reviewing 

results of those searches, Abt Associates will work with the EAD WAM to refine search criteria, and 

conduct additional data gathering efforts necessary to identify relevant, useful information, and fill gaps 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 16

identified in useful data sources. We anticipate that data gathering will continue throughout the period of 

performance of this contract.

In some instances, Abt Associates may conduct primary data collection activities to support decisionmaking. This support typically includes surveys of regulated entities to assemble detailed plant-specific 

data. Section 4 addresses primary data gathering.

2.4 Overall Data Quality Objectives and Criteria

EAD has determined that the data Abt Associates will be collecting and using in environmental and 

economic analyses to support the ELGs program represents influential scientific, financial or statistical 

information. As such, the Agency’s Information Quality Guidelines (EPA, 2002) establish a standard of 

quality for information, based on objectivity, utility, and integrity. Accordingly, this PQAPP describes 

how Abt Associates collects, manipulates, and analyzes data consistent with this standard of quality, and 

provides transparency regarding the data source, assumptions, methods, and statistical procedures.

The overall factors that Abt Associates uses in reviewing data for use in activities under this program

include transparency, accuracy, completeness, internal consistency, reasonableness, representativeness, 

and comparability. Below, we discuss considerations with respect to each of the three types of analyses.

2.4.1 Environmental Assessments

Figure 2-3 provides an example schematic summary of Abt Associates’ process for conducting 

environmental assessments in support of ELGs or other regulations. The diagram maps the program

phases (horizontal paths), key steps (boxes and arrow labels), and information flow (arrows). 

Figure 2-3: Process for Conducting Environmental Assessment.

INPUTS MODEL RUNS OUTPUTS

Industry Profile Model Selection

Model Setup, Calibration and Validation Environmental Assessment Endpoints Select model based on desired features (literature, experience, etc.) Gather data for setup, calibration and validation Inventory available models

Controls/loading reductions (e.g., from engineering analysis) Land use change or other effects

Hydrology

Water quality Human health risks Flow Water availability Human health effects Land-based effects

Baseline conditions

Review literature Gather data on baseline conditions Develop conceptual understanding of environmental effects of the policy

Translate scenario into model parameters Review/reduce outputs Extrapolate if needed

Design/Setup Phase Analysis Phase Reporting Phase

Environmental Assessment Report Describe baseline/policy conditions

Document model and uncertainty Summarize results Discuss implications

Identify relevant environmental indicators Highlight endpoints for benefits

Risks to aquatic and terrestrial wildlife

Preamble, ICR supporting statement and other documents

In-stream water quality and pollutant concentrations

Ecological risks To Benefits Analysis

The overall objective of the environmental assessment is to characterize and quantify the direct and 

ancillary effects of policies or regulatory actions on surface waters and other media. This assessment 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 17

generally involves compiling and analyzing data to describe baseline environmental conditions, and using 

models to evaluate how the policy or regulation may improve water quality, reduce human exposure to 

pollutants, and protect or enhance ecosystems. As shown by the arrow to the right of the flow diagram, 

the outputs from the environmental assessment also provide some of the inputs used to quantify and 

monetize the benefits of the regulatory action (see Section 2.4.3).

Abt Associates generally relies on existing data to support environmental assessments. These data 

characterize existing environmental conditions (effluent discharges, baseline concentrations), ecosystem 

or human health effects, and/or support development of models to assess changes in these conditions and 

effects due to the policy or regulation. We may obtain this data from the literature, environmental data 

sets, models, and other sources. We describe the procedures we use to gather, manage, assess, and analyze 

secondary data in Section 3 of this PQAPP. 

The following data quality objectives apply to existing data that Abt Associates uses for qualitative or 

quantitative environmental assessments:

 Reliability and accuracy: Abt Associates evaluates data for use in qualitative environmental 

assessments for reliability based on the apparent quality of the underlying scientific work as 

evidenced by the written source (e.g., published articles, databases, models). This evaluation 

involves verifying that the source identifies assumptions and methodologies, that sufficient 

information is available to evaluate the variability and uncertainty (quantitative and qualitative) in 

the information or in the procedures, measures, methods, or models, and that the assumptions and 

methodologies are consistently applied throughout the analysis as reported in the source. See 

Section 3.3 for details on applicable procedures. 

 Consistency and comparability: Source data need to be consistent across geographical and 

temporal dimensions relevant to an analysis. For national-level analyses, Abt Associates 

preferably selects data that are collected using consistent approaches across geographic locations

(states, counties, hydrologic regions). Abt Associates verifies that data collected to support 

environmental assessments are internally consistent and comparable to other similar data. This 

check involves confirming that the sum of individual elements equals reported totals, or 

comparing data sets and information to identify material differences. See Section 3.3 for details 

on applicable procedures.

 Completeness and representativeness: Abt Associates evaluates data for use in environmental 

assessment based on how well the conditions they describe represent the diversity of waterbodies, 

discharge sources, and pollutants addressed in the analysis. This evaluation includes considering 

the date when the information was collected, and differences in explanatory variables that may 

contribute to changes in observed conditions (e.g., changes in land use since data were collected 

or a model developed). See Section 3.3 for details on applicable procedures.

The objectives above drive not only our selection of the data used in environmental assessments, but also

the methods we employ to compile, manage, and analyze the data, as described throughout this PQAPP. 

For example, Abt Associates ensures accuracy of the information used in analyses by following data 

processing procedures that minimize the potential for transcription error, and by running validation

checks both within data set and against other sources. See Section 3.3 for details on applicable 

procedures.

2.4.2 Economic/Financial Analyses

Abt Associates conducts the following evaluations of the economic impacts of the ELGs program:

 Assessment of total social costs and benefits;

 Economic impact analyses (impacts on producers, consumers, etc.);
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 18

 Cost effectiveness analyses;

 Regulatory Flexibility Act (RFA) analyses;

 Unfunded Mandates Reform Act (UMRA) analyses; and

 Analyses to support executive orders:

 Executive Order 13563: Improving Regulation and Regulatory Review

 Executive Order 12866: Regulatory Planning and Review

 Executive Order 12898: Federal Actions to Address Environmental Justice in Minority 

Populations and Low-Income Populations

 Executive Order 13045: Protection of Children from Environmental Health Risks and Safety 

Risks

 Executive Order 13132: Federalism

 Executive Order 13211: Actions Concerning Regulations That Significantly Affect Energy 

Supply, Distribution, or Use

Figure 2-3 provides an example schematic summary of Abt Associates’ process for supporting EAD 

economic/financial analyses, including the program phases (horizontal paths), key steps (boxes and text 

arrows), data flow (arrows). Much of the data we use for these analyses are existing data (see Section 3), 

although we also collect primary data to support the analyses through surveys, as described in Section 4 of 

this QAPP.

Figure 2-4: Process for Conducting Economic/Financial Analyses.

INPUTS MODEL RUNS OUTPUTS

Industry Profile Definition of Actual/ Model Plant Owners Model Setup and Validation Impact Assessment Endpoints Build analytic framework Identify relevant indicators of economic achievability, economic impacts, etc. and applicable thresholds

Compliance technologies and costs Industry profile changes Plant/owner models Market-level model (e.g., IPM) Social analysis framework

Retirements/closures Cost-to-revenue ratios Impacts to government entities Cost-effectiveness and/ or cost reasonableness Pollutant loading reductions

Review literature Conduct industry survey Gather data on baseline conditions Compile data on technologies, plants, entities, etc.

Translate scenario into model parameters Review/reduce outputs Extrapolate if needed

Design/Setup Phase Analysis Phase Reporting Phase

Regulatory Impact Analysis / Economic Analysis Report Describe baseline/policy conditions

Document model and uncertainty Summarize results Discuss implications

Employment effects Distributional effects (small entities, households) Changes in energy production and use

Preamble, ICR supporting statement and other documents

Implementation schedule

Total compliance costs and total social costs

The following overall data quality objectives apply to data for economic/financial analyses:
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 19

 Accuracy and Reliability: Abt Associates evaluates existing data for use in economic/financial 

analyses for accuracy and reliability based on the primary purpose and scope of the data 

collection. For example, federal agencies devoted to the primary purpose of collecting and 

reporting economic data, including the Census Bureau, Bureau of Economic Analysis, and 

Bureau of Labor Statistics, meet federal data collection standards, and are among the most 

reliable, accurate, and consistent data for the economic analyses performed under this contract. 

Abt Associates uses the most current data available from these sources, and updates data to a 

common year using appropriate price or cost adjustment indices also available from these 

sources. Abt Associates fully documents the source, date, and any important considerations for 

comparability or gaps that the source may describe. These data cover measures of population, 

income, employment, wages, costs, and prices, by location, industry, and demographic 

characteristics. Abt Associates also utilizes data produced by comparable state-level agencies as 

best available. For data from all other sources (e.g., peer-reviewed journal, government-funded 

study, trade association study), Abt Associates evaluates the accuracy and reliability of the data, 

and discusses potential data quality issues with the EPA WAM. Abt Associates subsequently

documents the source, data, and rationale for use of particular sources and values, methods, and 

evaluations of the results of all data used in the deliverable document. Abt Associates also 

discusses potential biases, limitations, and uncertainties associated with the use of existing data in 

a separate section of deliverable reports. Abt Associates may use alternative sources to illustrate 

uncertainties.

 Transparency: The accompanying documentation must identify the primary purpose, sources, and 

methods for developing the values, and describe any exclusions (e.g., to prevent disclosure of 

confidential information) or adjustments (e.g., to report values in dollars of a given year). Where 

data involve estimation, the source must describe estimation methods, uncertainty-related issues, 

and the statistical uncertainty associated with the estimates.

 Reasonableness: The methods underlying development of values must be standard methods for 

the type of study. For example, technology costs could be based on modeling software that 

utilizes a work breakdown structure of individual component costs based on vendor quotes and 

industry sources (e.g., RS Means).

 Completeness and representativeness: Abt Associates uses data from a variety of sources to 

supplement or update the government and the peer-reviewed academic literature, such as the 

Engineering News-Record construction cost indices, and the Agency’s own databases of 

dischargers and discharges (Integrated Compliance Information System-National Pollutant 

Discharge Elimination System, ICIS-NPDES). These sources provide the most complete and 

representative information for specific data elements. Abt Associates evaluates the data using the 

criteria specified above, and fully documents the source, date, and any inconsistencies with other 

sources of similar information or across data from the source. Abt Associates discusses any data 

issues with the EPA WAM, and mitigation approaches, which may include primary data 

gathering to rectify the inconsistency or fill a data gap.

As for environmental assessments, these objectives drive our selection of data, and the methods we 

employ to compile, manage, and analyze the data. See Section 3.3 for details on applicable procedures.

2.4.3 Benefits Valuation

The overall objective of benefits valuation is to develop monetary values of the direct and ancillary 

effects of policies or regulatory actions on surface waters and other media. This analysis involves using 

outputs from the environmental assessment to identify human health, ecological, and economic 

productivity endpoints for which economic values exist or can be developed based on existing data, and 

estimating monetary values of benefits of the regulatory action. 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 20

Some of the benefits we expect from the ELGs and other regulations will manifest in economic markets 

through changes in price, cost, or quantity of market-valued activities. For benefits endpoints traded in 

markets, such as increased yields from commercial fisheries, we measure benefits by market prices or 

market-based factor pricing. We also use competitive prices to measure avoided cost type of benefits. For 

example, reduced pollutant loadings to public water supplies may lower costs of drinking treatment. We 

also use market prices to value direct medical costs of illnesses associated with exposure to pollutants. 

In other cases, benefits involve activities or sources of value that either do not involve economic markets 

or involve them only indirectly. We monetize these benefits, referred to as “nonmarket” benefits, based 

on a variety of market valuation approaches, including primary research methods (e.g., stated preference 

surveys), and benefit transfer from existing studies. 

In general, analyses of benefits from ELGs include the following benefit categories and approaches:

 Direct market use values (goods and products consumed or processed, such as commercial fish 

and shellfish), estimated using market-based approaches

 Nonmarket values (use values, such as recreation, aesthetic; nonuse values, including existence, 

aesthetic, bequest) estimated using benefits transfer, revealed preference approaches (including 

travel cost, hedonic pricing models, risk-wage approaches) or stated preference surveys

 Indirect use values (ecosystem services such as flood control, regulation of water flow and 

supply, and nutrient retention), estimated using productivity and cost -based approaches (e.g., 

effect on production; replacement costs; cost of providing substitute services; cost of avoided 

damages).

Figure 2-5 presents a schematic of Abt Associate’s process for conducting benefits analyses, using the 

specific example of the Steam Electric ELG, showing the linkage between the ecological endpoints

affected by the regulation and the quantification and monetization of benefits. (The category of 

impoundment-related effects specific to the steam electric industry may not be broadly applicable to all 

ELGs that EPA is developing.) The figure also distinguishes between the primary benefits, which are the 

environmental improvements that are the target of the rulemaking, and secondary benefits which may 

occur in conjunction with achieving the primary benefits. It is appropriate to account for secondary 

benefits in the regulatory analysis, although these benefits are not the primary objective of the regulatory 

action.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 21

Figure 2-5: Example Process for Conducting Benefits Analyses.

Abt Associates follows the general recommendations and principles for conducting benefit analyses 

outlined in EPA’s Guidelines for Preparing Economic Analyses (EPA 240-R-10-001; December 2010), 

including identifying the baseline, identifying potentially affected benefits categories, appropriate 

methods for estimating benefits, including conducting benefits transfers, and discounting future benefits.

To develop these analyses, Abt Associates primarily relies on existing data. These data characterize 

benefitting populations, society's willingness to pay (WTP) to avoid adverse health effects (e.g., cancer

and cardiovascular disease), the cost of an illness estimates (including medical costs, lost income, and 

other expenses ), WTP for improving ecosystem services and goods, consumer demand for market goods, 

production functions, replacement costs, cost of avoided damages, In addition, existing data can be used 

to support development of models or benefit valuation functions to assess changes in the relevant 

ecological or human health endpoints. 

We may obtain this data from the literature, economic data sets, models, and other sources (see Section 

3). In some (likely rare) instances, Abt Associates may collect primary data to support the analyses 

through surveys or focus groups (see Section 4). We may also develop a traditional (standalone) QAPP to 

cover new benefits method developments. With respect to overall data quality objectives for benefits 

analyses, the following apply:

 Accuracy and Reliability: Abt Associates utilizes data from a variety of studies that reflect 

methods and techniques accepted in the literature as capable of producing accurate and reliable 

information on the benefits of water quality and other environmental improvements. These 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 22

studies provide information on market and nonmarket amenity values, and related information for 

applying the values (e.g., measure of environmental change being valued, affected resource 

characteristics, availability of substitute sites, study methods, and other data). For amenity values, 

Abt Associates uses studies from the peer reviewed literature, and primary government studies 

subjected to peer review; for related information, data sources are federal agencies devoted to the 

primary purpose of collecting and reporting economic data. As described under economic and 

financial analysis, data from these sources meet federal data collection standards, and thus are 

among the most reliable, accurate, and consistent data for the economic analyses performed under 

this contract. Abt Associates uses the most current data available from these sources, and updates 

data to a common year using the appropriate price or cost adjustment indices also available from 

these sources. Abt Associates fully documents the source, date, and any important considerations 

for comparability or gaps that the source may describe. For data from the peer-review literature 

and government research, Abt Associates evaluates the accuracy and reliability of the data, and 

discusses potential data quality issues with the EPA WAM. Abt Associates subsequently

documents the source, data, and rationale for use of particular sources and values, methods, and 

evaluations of the results of all data used in the deliverable document. Abt Associates also 

discusses potential biases, limitations, and uncertainties associated with the use of existing data in 

a separate section of deliverable reports. Abt Associates may use alternative sources to illustrate 

uncertainties.

 Transparency: In order to use a source, the accompanying documentation must clearly identify 

the methods, data, and assumptions underlying the results.

 Reasonableness: Abt Associates evaluates the study results compared to results from other 

comparable studies for reasonableness. For example, benefit values may fall within the range of 

results estimated by other studies using different techniques, confirming the results. For estimates 

that fall outside the range estimated by other studies, we provide explanation of why the results 

make sense.

 Completeness and representativeness: Abt Associates uses studies that provide information on 

market and nonmarket, and uses and nonuse values, and representative of the environmental 

changes that the environmental assessment indicates will result from the rule. We supplement or 

update the values using cost and price indices as appropriate, as described for economic and 

financial analyses. These sources provide the most complete and representative information for 

specific data elements. Abt Associates evaluates the data using the criteria specified above, and 

fully documents the source, date, and any inconsistencies with other sources of similar 

information or across data from the source. Abt Associates discusses any data issues with the 

EPA WAM, and mitigation approaches, which may include primary data gathering to rectify the 

inconsistency or fill a data gap.

See Section 3.3 for details applicable procedures on data quality assessment and use.

2.5 Training and Certification

The Abt Associates staff, subcontractors, and consultants for this program have the requisite academic 

training and professional experience in the disciplines relevant to the areas they support: economics, 

finance, engineering, biology, chemistry, ecology, public policy, communications, and other areas. 

The Abt Associates Project Manager works with the Abt Associates Work Assignment Manager to ensure 

that work plans for individual WAs include the appropriate expertise and experience; the work plans 

document the proposed staffing plan. If EAD requires specialized skills not provided by the Abt 

Associates Team, the Abt Associates Project Manager will work with EAD to identify and obtain access 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 23

to the talent. The Abt Associates Quality Assurance Manager ensures that a qualified PQA Advisor 

supports each work assignment. 

Abt Associates will ensure that all staff involved in supporting this program receive a copy of this 

PQAPP, and of any work assignment-specific QAPP for projects they support. These staff will have 

received the requisite instructions in the QA/QC procedures, best practices, and tools.

2.6 Project-Level Documents and Records

The Abt Associates Program Manager and WAM are responsible for ensuring that project documents and 

records conform to technical requirements and quality system requirements specified in the WA or 

technical direction by the EPA WAM. The Abt Associates WAM sets up the overall file management 

structure for project files on the server, and instructs staff as to the proper organization of detailed files 

within that structure. We typically organize project file folders by task, analysis type or issue, and phase 

(e.g., proposed vs. final rule). Table 2-3 below provides an example of project folder organization. Abt 

Associates WAMs may modify the example to fit the particular project needs.

Table 2-3: Example project file management structure.

Folder Example subfolders and content

00-Background Detailed study

01-Admin Work plan and budget

Meeting notes

02-QAPP Quality Assurance Project Plan (QAPP)

Approval sheets

Audit findings

03-Survey Questionnaire

Information Collection Request (ICR)

Survey data

Quality Assurance (QA)

04-Cost and Economic 

Impact

Industry profile

Analyses

Report

05-Benefits Literature review

Analyses

Report

06-Technical Support Briefings

Preamble and rule text

07-Public Comments Comment summaries

Comment response document

For example, this PQAPP resides in a subdirectory of the contract-level directory on the Abt Associates 

server and is accessible to all staff working on the project. The Abt Associates Project Manager is 

responsible for reviewing the PQAPP at least annually to ensure that the content continues to be 

applicable to the project over time, and notifying the WAMs and PQAs of any updates. The Work 

Assignment Managers then notify project staff of the updates. The Work Assignment Manager is also 

responsible for notifying project staff of any updates to WA-specific QAPPs or SQAPPs. Updates to 

these documents reside in the project directories noted above, using a file naming convention that 

includes the date and version number (xx.xx.xx _v#) to signify revisions.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 24

At the project completion (or at other milestones determined by the Abt Associates WAM, PQA, or 

Project Manager), we archive and write-protect the work assignment file folder to prevent any changes to 

project files. In particular, we inventory files supporting a rulemaking record (proposal or final), and 

archive and/or write-protect the files to prevent changes. 

2.6.1 CBI

Abt Associates implements folder or file security measures as needed to protect sensitive project data. 

The Confidential Business Information (CBI) plan for Contract EP-C-13-039 (developed under this work 

assignment) describes our procedures for handling CBI, including information to be included in the 

record for a rulemaking.

2.6.2 Deliverables

The Abt Associates Program Manager and WAM are also responsible for ensuring that records and 

documents accurately reflect the completed work, and for ensuring that the project team archives files are 

upon completion of the project. Abt Associates produces a variety of deliverables under this contract, 

including:

 Draft and final reports (e.g., economic analysis report, environmental assessment report, benefits 

analysis report)

 Memoranda or issue papers describing methodologies, data sources, and assumptions

 Literature review summaries and databases

 Briefing slides or summaries

 Electronic files containing primary and existing data, analytic calculations, and results

 Electronic files (or hard copies if electronic versions are not available) of published literature, 

web pages, and other data sources

 Computer code and analytic framework, including programming comments and embedded 

metadata.

The work plans for each work assignment identify the deliverables to be produced, and format (e.g., 

report, spreadsheet, database) for each deliverable. The Abt Associates WAM works with the EPA WAM 

to identify the appropriate level of review for deliverables considering the type of document, audience, 

project phase, and sensitivity or complexity of content. For most projects, written reports – memoranda, 

issue papers, or formal reports – are a primary way to communicate the results of an analysis to EPA and 

to the public. As such, Abt Associates follows QA procedures to ensure that such written reports support 

the project’s data quality objectives, and present the data, methodology, results, and limitations or sources 

of uncertainty accurately, clearly, and with details sufficient to allow the results to be reproduced. 

When determining the QA review process of written documents, Abt Associates categorizes written 

reports into three groups, shown in Table 2-4. Note that the QA review described in this table refers 

specifically to the written document, not the details of the content, which will have been reviewed already 

as part of the underlying analysis or research task.

Table 2-4: QA review categories for written deliverables.

Level Examples QA Review

1 Scoping memorandum

Summary table

Briefing slides

Outline review by WAM (and PQA Advisor if appropriate).

Detailed review by author and peer (or WAM if appropriate).

Numbers cited in the document are accurate, as verified in 

the underlying sources.

Principal sources are cited.

Principal limitations are noted.
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 25

Table 2-4: QA review categories for written deliverables.

Level Examples QA Review

Document is free of grammatical errors.

Document uses established formatting style.

2 Draft analytic blueprint

Draft chapters for final report

Level 1 plus:

Detailed review by WAM for accuracy, clarity of 

presentation, and completeness.

All sources are cited.

Limitations and biases are described.

Optional review by editor.

Optional formatting review.

3 Economic Analysis report Level 2 plus:

Editorial and formatting review (clarity of presentation, 

consistency of formatting, etc.).

Final acceptance by Project Manager or his designee, 

including verifying use of appropriate language and 

characterization of the results.

PQA = Project Quality Assurance

QA=Quality Assurance

WAM = Work Assignment Manager

The Abt Associates WAM reviews all written deliverables (memoranda, reports, issue papers, etc.), as 

appropriate, and identifies a qualified reviewer for material he/she authored. The reviewer assesses the 

clarity and accuracy of the written document, including verifying that the document has a clearly 

identified purpose; the findings are clear and supported by the information presented; and that any 

analysis results and data match information from the underlying analytic files (spreadsheets, program 

outputs, etc.).

For longer reports targeted at the public and non-EAD audiences, a skilled technical editor conducts a 

final editorial review. The editor’s role is to verify correct grammar, spelling, and text organization/flow, 

as well as to ensure that the style and flow of the document are appropriate for the intended audience. 

Upon direction by the EPA WAM, Abt Associates facilitates external peer review of the methodologies 

and finished products.

After Abt Associates’ internal technical reviews have been completed (and the EPA WAM’s reviews as 

well, as appropriate), the relevant project management staff – usually the individual content area 

managers – address all comments and revisions. For major deliverables and final reports, such as reports 

that will be part of the rulemaking record, the Abt Associates Project Manager or designee will provide an 

additional final acceptance before submittal to EPA. Individual reviewers complete the applicable 

signature forms upon completion of the requisite review items. Appendix A provides the deliverable 

review and signature forms/checklists that we will use for this contract.

Throughout the performance of the work, Abt Associates project staff maintain a log of substantive team 

discussions about the analyses, data issues, and decisions that affect the project. We also use this log (or 

different logs for specific aspects of a project) to document the outcome of discussions with the EPA

WAM that result in modifications to the analysis, and note any necessary follow-ups. 

2.6.3 Quality Audits

On a quarterly basis, the Abt Associates Project Manager (or his designee) will review implementation of 

this PQAPP under select work assignments. This QA review/audit, which will involve the WAM, PQA

Advisor, and project staff if applicable, will include discussion of ongoing QA/QC procedures, 
Project Management 

Abt Associates PQAPP EP-C-13-039 July 2014 26

documentation of such procedures, and discussion of any issues encountered in the performance of the 

work, along with any mitigation measures, needed adjustments to QA/QC procedures, or other actions to 

be implemented by the project team. The Project Manager (or his designee) will record performance and 

outcome of the quarterly review using the form depicted in Appendix B. The WAM will inform the EPA

WAM and/or Project Manager of the status of any follow-up items identified during the meeting.

The WAM will document QA/QC procedures, and any adjustments to the procedures, in the monthly 

progress reports, as described in Section 3.3.5. The Project Manager will address any identified internal

needs, such as training, through means provided by the company (e.g., professional development plans for 

staff, brown bag seminars).
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 27

3. Existing Data

This section describes Abt Associates’ procedures for data 

gathering, management, QA, and use pertaining to existing data.

3.1 Data Gathering

Abt Associates’ support to EAD for the ELG program primarily 

involves use of existing data to answer analytical questions. We 

may use existing data exclusively, or in combination with primary data, to provide timely and effective 

basis for conducting the range of analyses under this project. Since we often obtain these data from 

publicly available sources, existing data may also provide transparency to the analyses, which is one of 

the overall data quality objectives discussed in Section 2.4. 

Abt Associates works with the EPA WAM to define data needs for the work assignment, considering the 

intended use of the data. EPA defines these needs in the work assignment PWS; subsequently, Abt 

Associates discusses the needs in the technical approach described in our work plan, meetings and 

discussions, and planning memoranda detailing our understanding of the analytic issues and potential 

approaches. 

The following sections describe our process for gathering data, differentiating across the three types of 

analyses where appropriate. 

3.1.1 Data Needs

Table 3-1 provides a detailed description of common data needs for the three types of analyses conducted 

under this contract, and sources for those data, consistent with the overall objectives discussed in Section 

2.4, and based on ongoing and recent analyses performed for EAD. For each data source, we also 

summarize the rationale for selecting the data, consistent with the criteria discussed in Section 3.1.3.

Table 3-1 summarizes the publicly available existing data sources that we use in our analyses. Abt 

Associates also uses data generated by EPA or its other contractors to support a specific rulemaking. 

Examples of such data include outputs from engineering analyses (technology costs, loading reductions, 

etc.) that we use as inputs in our economic impacts, environmental assessment, or benefits analyses. We 

manage these data similarly to “secondary data” in the sense that they are not data we obtained or 

generated internally. However, we do not apply the same level of applicability review as other obtained

secondary data because EAD has already deemed the quality of these data to be acceptable through 

comparable PQAPPs, SQAPPs, or QAPPs associated with the contracts under which such data is 

obtained. 

Existing Data Sources

 Literature

 Published databases

 Existing models
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 28

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

Land Cover and Land Use Data

U.S. Geological Survey 

(USGS) Hydrologic Unit 

Code (HUC) geospatial 

data

Watershed hydrologic 

resources and boundaries

Estimating the effect of 

regulatory options; identifying 

affected sources of pollution

Federal government source.

Standard geographical unit 

definitions used across federal 

government agencies.

http://water.usgs.gov/GIS/h

uc.html

Environmental Protection 

Agency (EPA) NHDPlus 

geospatial data

Geo-spatial, hydrologic 

framework dataset

Modeling the effect of regulatory 

options

Federal government source.

Standard geographical unit 

definitions used across federal 

government agencies.

Publicly available and widely used.

http://www.horizonsystems.com/nhdplus/

Google Earth Desktop mapping software that 

shows current and (where 

available) recent historical 

satellite images 

Reviewing, viewing, and 

collecting spatial data on land 

and water features; quickly 

viewing land-use and landcover change over time

Global maps and spatial data which 

are publicly available, widely used, 

and free. Recent images are 1-3 

years old and updated periodically. 

Images have up to 1-meter 

resolution in many areas.

Can be used to develop shapefiles 

for analysis in GIS software.

http://www.google.com/eart

h/explore/products/desktop.

html

Integrated Climate and 

Land Use Scenarios 

(ICLUS)

Data associated with scenarios 

broadly consistent with globalscale, peer-reviewed storylines 

of population growth and 

economic development, which 

are used by climate change 

modelers to develop 

projections of future climate

Projecting future land use and 

impervious surface cover based 

on assumptions about 

population growth, 

transportation patterns

Federal government source. 

National-level data developed 

using consistent methodology, 

agreed upon assumptions.

Publicly available.

http://www.epa.gov/ncea/gl

obal/iclus/

National Land Cover 

Data (NLCD)

Consistent land cover data 

applied across the 

conterminous United States at 

a spatial resolution of 30 

meters

Identifying land types and 

potential pollutant sources

Federal government source. 

Most comprehensive national-level 

dataset.

Publicly available and widely used.

http://www.mrlc.gov/nlcd20

06.php
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 29

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

U.S. Forest Service 

Urban Forest Data and iTree Data

Databases and publications, 

including city-specific 

analyses, addressing urban 

forest services including rates 

of tree canopy cover, human 

health, environmental quality 

and energy savings

Estimating ancillary benefits of 

reductions in impervious 

services (i.e., increases in 

green space)

Federal government source.

Publicly available and widely used.

National-level datasets. 

http://www.nrs.fs.fed.us/dat

a/urban/; 

http://www.itreetools.org/

Ecological and Health Risk Data

Fish and Wildlife Service 

(FWS) Threatened and 

Endangered Species 

System database

Provides a list of species 

designated as threatened or 

endangered and designated 

critical habitat areas

Identifying threatened and 

endangered species that may 

be affected by regulatory 

options

Federal government source.

Most comprehensive national-level 

dataset for data.

Publicly available.

http://ecos.fws.gov/ecos/ho

me.action

EPA Integrated Risk 

Information System 

(IRIS)

Information on health effects 

from exposure to 

environmental contaminants

Identifying endpoints, risk 

values and key studies for 

estimating baseline human 

health risks, and risk reductions 

from regulatory options; key 

values include unit risk 

estimates, or regression 

equations from primary studies

Federal government source. 

National-level dataset collected 

using consistent methodology over 

time and across industries.

Subject to extensive scientific / QA 

review prior to publication.

Publicly available and widely used.

http://www.epa.gov/iris/

Agency for Toxic 

Substances and Disease 

Registry (ATSDR) 

Database

Toxicological profile for 

pollutants

Identifying endpoints, risk 

values and key studies for 

estimating baseline human 

health risks, and risk reductions 

from regulatory options; key 

values include unit risk 

estimates, or regression 

equations from primary studies

Federal government source. 

Peer-reviewed summary of the 

toxicological and adverse human 

health information on several 

contaminants. Publically available.

http://www.atsdr.cdc.gov/to

xprofiles/index.asp
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 30

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

National Toxicology 

Program (NTP)

Frequently updated Report on 

the Carcinogens

Reviews of adverse health 

effects for other contaminants

in NTP Monographs

Identifying endpoints, risk 

values and key studies for 

estimating baseline human 

health risks, and risk reductions 

from regulatory options; key 

values include unit risk 

estimates, or regression 

equations from primary studies

Federal government source. 

Systematic classification of the 

Weight-of-Evidence for 

carcinogenicity 

NTP Monographs are detailed, 

peer reviewed summaries of the 

evidence for adverse health effects

http://ntp.niehs.nih.gov/ntp/

roc/twelfth/roc12.pdf;

http://ntp.niehs.nih.gov/?obj

ectid=497C419D-E834-

6B35-8AF15D389859AF07

California EPA Provides several peer 

reviewed risk values; may be 

updated more recently than 

IRIS or other EPA sources

Supplementing risk values for 

endpoints identified through 

IRIS, ATSDR or NTP; key 

values include unit risk 

estimates, or regression 

equations from primary studies

State government source. 

Scientifically sound, peer reviewed 

risk estimates

http://oehha.ca.gov/about.h

tml

International Agency for 

Research on Cancer 

(IARC)

Provides peer reviewed 

weight-of-evidence 

classifications for carcinogens

Supplementing information from 

IRIS, ATSDR and NTP

Scientifically sound, peer reviewed, 

internationally known weight-ofevidence classifications

http://monographs.iarc.fr/E

NG/Classification/

Water Quality and Discharge Data

EPA water quality criteria 

and maximum 

contaminant level (MCL)

Ambient and drinking water 

levels of constituents 

protective of aquatic life, 

human health, and aquaticdependent wildlife

Estimating environmental 

benefits of regulatory options

Federal government source. 

Regulatory/statutory values.

Publicly available.

http://water.epa.gov/scitech

/swguidance/standards/crit

eria/current/index.cfm; 

http://water.epa.gov/drink/c

ontaminants/

USGS National Water 

Information System 

(NWIS)

Water resources data from

approximately 1.5 million sites

nationwide

Assessing baseline pollutant 

concentrations and benefits of 

regulatory options

Federal government source. 

National-level dataset collected 

using consistent methodology over 

time and across industries.

Publicly available and widely used.

http://waterdata.usgs.gov/n

wis
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 31

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

EPA’s Watershed 

Assessment, Tracking & 

Environmental Results 

System (WATERS)

Water quality information 

including designated uses, 

water quality assessment and 

monitoring results, causes and 

sources of impairments, public 

beach closures, and 

discharger locations 

Assessing baseline 

environmental conditions and 

benefits of regulatory options

Federal government source.

Most comprehensive source of 

information on water quality 

assessments. 

Incorporates verified information 

from other sources (states).

Publicly available.

http://www.epa.gov/waters/i

r/

EPA’s Safe Drinking 

Water Information 

System (SDWIS)

Information on drinking water 

systems, including 

characteristics, monitoring 

results, violations, and 

enforcement actions 

Identifying environmental risks 

to drinking water systems, and 

benefits of regulatory options

Federal government source.

Comprehensive national-level 

source for drinking water treatment 

systems.

http://water.epa.gov/scitech

/datait/databases/drink/sdw

isfed/

EPA’s Integrated 

Compliance Information 

System-National 

Pollutant Discharge 

Elimination System 

(ICIS-NPDES)29

Information on permitted 

discharges to surface waters

Characterizing the types and 

quantities of pollutants 

discharged to waters by NPDES 

permittees; assessing 

compliance with permitted limits

Federal government source.

Most comprehensive national-level 

dataset for a NPDES permits.

Publicly available and widely used.

http://cfpub.epa.gov/npdes/

npdesnoi.cfm

USGS Reservoir 

Sedimentation Survey 

Information System 

(RESIS)

The most comprehensive 

compilation of data from 

reservoir sedimentation 

surveys throughout the 

conterminous United States 

Evaluating baseline reservoir 

sedimentation rates

Federal government source.

National-level datasets. 

Publicly available.

Most comprehensive source of 

reservoir sedimentation survey 

data.

http://pubs.usgs.gov/ds/ds4

34/

Air Quality and Emissions Data

Meteorological Data from 

EPA’s Support Center for 

Regulatory Atmospheric 

Modeling (SCRAM)

Surface and upper air 

meteorological data from

stations across the United 

States

Estimating mixing heights for 

converting mass of air pollutant 

removal to changes in 

concentrations

Federal government source.

Used in the relevant literature.

Publicly available.

http://www.epa.gov/ttn/scra

m/metobsdata_databases.

htm
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 32

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

U.S. EPA Emissions and 

Generation Resource 

Integrated Database 

(eGRID)

Database of environmental 

characteristics of power 

generation in the U.S. 

including air emissions, 

emissions rates, net 

generation, resource mix, and 

other attributes

Estimating the benefits of 

changes in energy usage under 

regulatory options

Federal government source.

Publicly available and widely used. 

National-level datasets.

http://www.epa.gov/cleanen

ergy/energyresources/egrid/index.html

Company Financial Data

Financial Reports to the 

US Securities and 

Exchange Commission

Qualitative and Quantitative 

information on the reporting 

company’s financial condition

Characterizing the financial 

condition of a company and/or 

industry category

Federal government source.

Official reports to meet statutory 

regulatory requirements

http://www.sec.gov/about/a

nnrep.shtml

Federal Energy 

Regulatory Commission 

(FERC) Form 1/1-F data

Comprehensive financial and 

operating report submitted for 

Electric Rate regulation and 

financial audits

Characterizing the financial and 

operating conditions of electric 

utilities

Federal government source.

Official reports to meet statutory 

regulatory requirements

http://www.ferc.gov/docsfiling/forms.asp#1F

Value Line Investment 

Analyzer

Proprietary financial data for 

over 1,000 publicly traded 

firms

Characterizing the financial 

condition of a company and/or 

industry category

Contains information for 250 data 

fields, on approximately 1,700 

major stocks, representing nearly 

100 industries.

10 years of data history.

http://www.valuelinepro.co

m/Services/Equity/TheValue-Line-InvestmentAnalyzer/

Dunn & Bradstreet Company-level data including 

NAICS code, employment, 

location, revenue and 

ownership type

Characterizing the financial 

condition of a company and/or 

industry category and 

characterizing baseline financial 

condition for use in analyses

Largest commercial database 

compiled thousands of sources.

Subject to extensive and rigorous 

QA procedures.

http://www.dnb.com

EPA’s Electronic Notice 

of Intent (eNOI)

Information on sites/facilities 

that apply for coverage under 

selected general permits 

(Construction General Permit 

(CGP), Multi-Sector General 

Permit (MSGP), Pesticides 

General Permit (PGP) and 

Vessel General Permit (VGP))

Characterizing facilities 

potentially subject to regulatory 

options

Federal government source.

National-level dataset for a various 

types of general permits.

Publicly available and widely used.

http://cfpub.epa.gov/npdes/

npdesnoi.cfm
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 33

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

Industry Financial Data

US Census Bureau

Economic Census

Census of economic and 

employment data by NAICS 

and size categories, conducted 

every 5 years for all industrial 

sectors except the agricultural 

and mining sectors

Estimating the number of 

entities in an industry category; 

characterizing baseline 

economic/financial conditions 

(employment, revenue, trends)

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across economic sectors.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://www.census.gov/eco

n/

US Census Bureau

Quarterly Financial 

Report

Quarterly financial data for 

manufacturing, mining , trade 

and selected service industries

Characterizing the financial 

performance and condition of 

an industry category

Federal government source.

Official government source for the 

data. 

Publicly available and widely used.

http://www.census.gov/eco

n/qfr/

Board of Governors of 

the Federal Reserve 

System, Economic & 

Research Data

Industrial production and 

capacity utilization data by 

selected industry sectors

Characterizing the financial 

condition of an industry 

category

Federal government source.

Official government source for the 

data. 

Publicly available and widely used.

http://www.federalreserve.g

ov/econresdata/default.htm

US International Trade 

Commission, Interactive 

Tariff and Trade 

Dataweb6

Data on US tariffs and 

international trade

Characterizing import 

competition and export 

dependence for an industry 

category

Federal government source.

International-level dataset collected 

using consistent methodology over 

time and across industrial sectors.

Publicly available and widely used

http://dataweb.usitc.gov/

The Risk Management 

Association, eStatement 

Studies

Proprietary financial data by 

NAICS code

Characterizing the financial 

condition of an industry 

category

Data comes directly from the 

financial statements of institutions.

15 years of data history

http://www.rmahq.org/toolspublications/tools/estateme

nt-studies

Standard & Poor’s 

Industry Surveys and 

Stock Reports

Financial information for 

selected companies and 

overall industry, including 

rating values (e.g. debt rating) 

and industry outlooks 

Characterizing the financial 

performance and condition of 

an industry category

Tracks data for 50 North American 

industries and 10 Global industries.

Trend analysis by top economists.

http://www.netadvantage.st

andardandpoors.com/
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 34

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

Other Industry Data

Small Business 

Association (SBA)

Business size standards, by 

industry sector

Identifying small entities Federal government source.

Regulatory/statutory values.

Publicly available.

http://www.sba.gov/categor

y/navigationstructure/contracting/contra

cting-officials/eligibility-sizestandards

US Census Bureau 

Census of Governments

Census of all state and local 

government organization units

conducted every 5 years

Characterizing baseline public 

finance and employment

conditions; comparing 

compliance costs to receipts

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across jurisdictions.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://www.census.gov/gov

s/

U.S. Energy Information 

Administration Electricity 

Data Browser

Data on the generation, sale, 

and consumption of electricity

Characterizing baseline 

operations, and 

economic/financial conditions of 

the electric power sector

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across industries.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://www.eia.gov/electricit

y/data/browser/

U.S. Energy Information 

Administration Annual 

Energy Outlook

Yearly projections and analysis 

of energy topics

Supporting analyses of 

economic impacts of regulatory 

options (e.g., estimating future 

energy demand and cost) and 

benefits (e.g., estimating 

benefits of energy savings)

Federal government source.

Meets specific data needs

Based on best available data.

Publicly available.

http://www.eia.gov/forecast

s/aeo/

U.S. Energy Information 

Administration, Other 

Data sets

Various data sets on the 

production, prices, disposition 

of energy (electricity, coal, 

natural gas, petroleum, 

renewables, etc.), and energy 

consumption

Characterizing baseline 

operations, and 

economic/financial conditions of 

the energy sector 

Federal government source.

Meets specific data needs.

Publicly available.

http://www.eia.gov/
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 35

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

USGS Minerals 

Yearbook

Annual statistics on material

and mineral industries 

(production, consumption, 

etc.), economic and technical 

trends, and development

Characterizing baseline 

operations, and 

economic/financial conditions of 

the material and mineral 

industries

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across industries.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://minerals.usgs.gov/mi

nerals/pubs/myb.html

USGS Minerals 

Commodity Summaries

Annual statistics on minerals 

and materials domestic 

industry infrastructure, 

government programs, tariffs 

and statistics

Characterizing baseline 

operations, and 

economic/financial conditions of 

the material and mineral 

industries

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across industries.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://minerals.usgs.gov/mi

nerals/pubs/mcs/

Grubb & Ellis, Inc. Reed 

Construction Data 

(RSMeans)

Information on the 

characteristics of construction 

projects (e.g., size and value)

Characterizing commercial/ 

industrial and residential 

construction projects; estimating 

the economic impact of 

regulatory options

Accurate and up-to-date cost 

information for thousands of items.

http://rsmeans.reedconstru

ctiondata.com/

IHS Economics and 

Country Risk

Construction Market 

Analysis and Forecasts

Construction market analysis 

and forecast

Characterizing baseline 

operations, and 

economic/financial conditions of 

the construction sector

Insight from leading 

macroeconomic experts on the 

construction industry.

Covers U.S. and global markets.

Based on data from U.S. Census 

Bureau.

http://www.ihs.com/product

s/globalinsight/industry/constructio

n/index.aspx

National Oceanic and 

Atmospheric Association 

(NOAA) Commercial 

Landings Statistics

Annual and monthly data on 

commercial fish landings (in 

metric tons/pounds and 

dollars) by species and 

geographical area

Estimating commercial fishing 

values

Federal government source.

National-level dataset collected 

using consistent methodology over 

time.

Clear description of data caveats 

available.

Publicly available.

http://www.st.nmfs.noaa.go

v/st1/commercial/
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 36

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

NOAA’s National Marine 

Fisheries Service 

(NMFS) Marine 

Recreational Fishing 

Statistics Survey

Nearly 30 years of data on 

marine recreational catch, 

effort, and participation 

throughout the world

Estimating the impact of 

recreational fishing on marine 

resources

Federal government source.

National-level dataset collected 

using consistent methodology over 

time. 

Standardized data.

Scientifically credible and 

consistent data.

Publicly available.

http://www.sefsc.noaa.gov/

about/mrfss.htm

Property Transaction 

Records and 

Characteristics Files

State- level databases 

(Maryland, New York, and 

others) of real estate 

transaction records, also 

including state-level 

aggregations of property 

assessment data on structural 

and parcel characteristics

Conducting hedonic property 

value analyses of the value of 

environmental amenities and 

other parcel-level analyses 

(commercial or industrial 

analyses; sprawl analysis)

State government source.

Need for state-specific data based 

on current models (New York, 

Maryland).

Data are aggregated from standard 

property transfer and assessment 

forms submitted at local 

jurisdictions.

Subject to QA review by state, 

county, and/or municipal agencies.

Availability depends on specific 

data source. For example, NY data 

are free with a FOIL request. MD 

data are not free.

http://www.tax.ny.gov/resea

rch/property/online/intro.ht

m

http://swcf.tax.ny.gov/sales

web/

http://planning.maryland.go

v/OurProducts/PropertyMa

pProducts/MDPropertyView

Products.shtml
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 37

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

Parcel Map Layers GIS dataset of point features 

for real property tax parcels by 

county or state (New York, 

Maryland, and others). These 

points represent the visual 

center of each parcel on a tax 

map

Conducting hedonic property 

value analyses of the value of 

environmental amenities and 

other parcel-level analyses

State government source.

Need for state-specific data based 

on current models (New York, 

Maryland).

State-level dataset of parcel point 

layers, including map layer 

versions from recent history.

Can be readily merged with other 

data, including other GIS layers 

and property transaction records 

from the relevant state.

Availability depends on specific 

data source. For example, NY data 

are free with a FOIL request. MD 

data are not free.

http://gis.ny.gov/gisdata/inv

entories/results.cfm?themeI

Ds=2

http://planning.maryland.go

v/OurProducts/PropertyMa

pProducts/MDPropertyView

Products.shtml

Population and Demographic Data

US Census Bureau 

Decennial Census, State 

& County QuickFacts,

and American 

Community Survey

Census of population and 

socioeconomic data conducted 

every ten years, and ongoing 

statistical survey that samples 

a percentage of the population 

every year

Modeling and analysis, 

including environmental justice, 

of population, housing, and 

other socio-demographic 

characteristics; benefits 

valuation functions

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across economic sectors.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://www.census.gov/201

0census/; 

http://quickfacts.census.gov

/qfd/index.html#;

https://www.census.gov/ac

s/www/

Woods and Poole 

Economics Inc. 

Database with projected 

economic and demographic 

data at the state, region, 

county, and metropolitan area 

levels through 2040

Adjusting future populations to 

account for population changes 

in benefits analyses

Meets specific data needs.

National-level dataset.

Updated annually.

http://www.woodsandpoole.

com/
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 38

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

Valuation and Economic Impact

Literature on the Value of 

Improving Aquatic and 

Terrestrial Ecosystem 

Services and Goods 

Studies in the literature that 

contain estimates of 

willingness to pay (WTP) for 

surface water quality 

improvements, reduced risk to 

aquatic, terrestrial and avian 

species, changes in open and 

vegetation

Developing a WTP value or a 

valuation function

See Section 3.1.3 See Appendix C

Interagency Working 

Group on Social Cost of 

Carbon

Report by the U.S. government 

interagency working group on 

social cost of carbon 

presenting values by year 

under various discount rates

Estimating the benefits of 

changes in greenhouse gas 

emissions under regulatory 

options

Federal government source.

Publicly available and widely used. 

Interagency report.

http://www.whitehouse.gov/

sites/default/files/omb/infor

eg/social_cost_of_carbon_f

or_ria_2013_update.pdf

FWS National Survey of 

Fishing, Hunting, and 

Wildlife-Associated 

Recreation

Data on participation in 

hunting, fishing, and wildlife 

viewing by region; 

Quantification of the economic 

impact of wildlife-based 

recreation

Estimating economic impacts 

and benefits of regulation 

affecting wildlife-based 

recreation

Federal government source.

National-level and state-level data.

Data collected by U.S. Census 

Bureau computer-generated 

interviews (standardized).

Publicly available.

http://wsfrprograms.fws.gov

/Subpages/NationalSurvey/

National_Survey.htm

Practice Data

International Stormwater

Best Management 

Practice (BMP) Database

Database of over 530 BMP 

studies, performance analysis 

results, tools for use in BMP 

performance studies, 

monitoring guidance and other 

study-related publications

Evaluating performance of 

modeled stormwater BMPs (i.e., 

validate results obtained using

the Soil and Water Assessment 

Tool (SWAT) or other models)

Sponsored by EPA and industry 

groups.

Most-comprehensive and mostcited database of BMP information.

http://www.bmpdatabase.or

g/

U.S. Army Corp of 

Engineers (USACE)

Dredging Information 

System

Database of USACE and 

USACE contracted dredging 

jobs include volume dredged, 

data, location, and cost

Estimating benefits of reduced 

sedimentation and reduced 

dredging under regulatory 

options

Federal government source.

National-level datasets.

Publicly available.

Detailed source for both volumes 

and costs.

http://www.navigationdatac

enter.us/dredge/dredge.ht

m
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 39

Table 3-1: Detailed Description of Existing Data.

Data Source Description Typical Data Need/Use Criteria for Selecting Resource

Stormwater Management 

Manuals

State-specific manuals that 

specific design criteria for 

various stormwater practices

Estimating the amount of each 

vegetation type in “acres of 

vegetation per acre of BMP”

Need for state-specific data.

Publicly available.

Varies based on state.

Indices

Bureau of Economic 

Analysis GDP deflator

Index of inflation Updating costs and benefits to 

common year values

Official federal government source 

for the data. 

Publicly available and widely used.

http://www.bea.gov/

Bureau of Labor 

Statistics Analysis 

Consumer Price Index

(CPI)

Monthly data on price changes 

for a representative basket of 

goods and services, 

purchased by urban 

consumers

Updating costs and benefits to 

common year values

Official federal government source 

for the data. 

Publicly available and widely used.

http://www.bls.gov/cpi/

Engineering NewsRecord Construction 

Cost Index (CCI)

Construction cost index Updating construction- and 

engineering operations-related 

costs/prices to common year 

values

Based on actual prices reported 

from the same suppliers in 20 U.S. 

cities

Widely used.

http://enr.construction.com/

economics/

Bureau of Labor 

Statistics, Employment

Cost Index and wages

(ECI)

Inflation index for wages and 

benefits by region and sector

Updating components of 

costs/prices (except 

construction and non-labor 

operating costs) to common 

year values; estimating labor 

costs associated with burden 

hours

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across industries.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://www.bls.gov/data/

Bureau of Labor 

Statistics, Producer Price 

Index

Index of the average change in 

output prices received by 

domestic producers 

Characterizing product pricing 

in an industry category 

Federal government source.

National-level dataset collected 

using consistent methodology over 

time and across industries.

Subject to extensive QA review 

prior to publication.

Publicly available and widely used.

http://www.bls.gov/ppi/
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 40

3.1.2 Potential Data Sources

Abt Associates relies on the existing data listed in Table 3-1 and other data generated specifically for the 

analysis by EPA or other contractors (under separate QAPPs) to characterize economic or environmental 

conditions that may change because of the regulatory options (and support evaluation of the changes), and 

to provide key values for analytic frameworks. If we determine that these data are not sufficient to support 

the analyses, we will identify additional data sources and describe these sources in a supplemental QAPP 

or amend this PQAPP. We describe the process for identifying potential data sources below.

Sources of additional existing data include academic journals and other peer-reviewed publications, 

government reports and databases, and working papers. Abt Associates will identify additional data 

sources through online searches (see Literature section below for discussion of search tools and key 

words), and bibliographies or publications from government organizations (see below for a list of 

sources). Abt Associates compiles data used on prior projects done for EAD and other clients, and 

continuously reviews relevant literature on economic and environmental data to support the 

environmental and economic regulatory analyses under this program. For example, Abt Associates will 

use its experience from prior regulatory impact analyses and environmental assessments, Internet

searches, and personal communications to identify estimates of unit costs and benefits from water 

pollution controls and water quality improvements specific to a given industry.

We discuss our approach for conducting research of existing literature, published databases, and models

below.

Literature

Abt Associates uses searches of the following to identify relevant literature:

 General literature databases and search engines (EBSCOhost, Google Scholar, Google)

 Topic-specific reference and abstract databases such as

 benefit valuation literature: Environmental Valuation Resource Inventory (EVRI), Benefits 

Use Valuation Database (BUVD), AgEcon Search, EconLit, and RePEc/IDEAs,

 Soil and Water Assessment Tool (SWAT) database compiled by Texas A&M University,

 health effects; PubMed and TOXNET

 Webpages of authors and university programs known to publish relevant research (e.g., on stated 

preference studies and/or water quality valuation)

 Web sites of organizations and agencies known to conduct relevant research (e.g., Resource for 

the Future, EPA National Center for Environmental Economics for research on resource 

economics valuation; U.S. Geological Survey (USGS) and EPA Office of Research and 

Development for research on water quality impacts of pollutants).

 Key resource economics journals (Land Economics, Environmental and Resource Economics, 

Marine Resource Economics, Journal of Environmental Economics and Management, Water 

Resources Research, Ecological Economics, Review of Environmental Economics and Policy, 

and Journal of the Association of Environmental and Resource Economists).

When conducting literature reviews, we identify potentially relevant articles and reports based on title, 

keywords, and abstracts, either by directly searching electronic publication databases or websites or by 

reviewing the literature cited in other relevant sources. We identify search terms from a preliminary list of 

relevant topics for the particular analysis or type of impact. Based on results of the searches, we then 

either expand the terms to be more general or refine them to get more specific entries (e.g., searches for 

“stormwater AND runoff AND impacts” may be refined to focus on “stream erosion” instead of general 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 41

impacts; ensuring the correct chemical form such as Hg versus MeHg). For example, we may identify 

other terms often associated with the desired keywords from articles that seem relevant upon initial 

review.

We then review the abstracts to screen relevant articles or reports and, if warranted by the breadth of the 

research and number of articles, categorize the articles and reports according to topics and subtopics. To 

the extent possible, we prioritize literature from peer-reviewed journals and government reports. We then 

carefully review sources that pass this screening step, and summarize and synthesize information in a 

short write-up or tabular format. We review the documented methods for rigor of study design, methods, 

and results. 

Abt Associates staff identifying and reviewing the literature have training in the disciplines pertaining to 

the research topics – economics, engineering, environmental sciences – and are qualified to assess the 

relevance of the data, quality of the analysis, and appropriateness of the results and conclusions against 

the objectives. If we reject a relevant article as a result of the review, we document the reason for this 

rejection in the accompanying notes. 

For example, the following describes the literature search process for recent ELGs.

 Step 1: We work with the EPA WAM to identify the issues to be covered by the review. This step 

may include developing a method memorandum that includes the proposed outline for the 

eventual report that will summarize the findings of the review, and identifying the search 

parameters (search tools, period of interest, search terms, review/reduction process). Appendix C

lists the search terms we are using for two meta-analyses.

 Step 2: We accumulate abstracts, papers, and data sources in a reference database for more 

careful review.

 Step 3: We periodically review interim findings with the EPA WAM, noting progress with 

identifying sources for the various outline topics and apparent gaps.

 Step 4: We review and categorize sources in accordance to relevance to the outline topics. We 

also include metadata, and data to help assess the quality of the sources: EndNote DCN; 

Reviewer and Date; Authors; Year; Title; Additional Citation Information; Length; Resource type 

(Journal Article/Report/Information Request/etc.); Source (Government, Federal/Government, 

State/Academic/Industry/etc.); Peer Reviewed (Yes/No/Uncertain); Bias (None/Unclear/AntiSGE/Pro-SGE); Not Relevant? (Flag); Notes; Internal Notes; Flag for relevant topics (based on 

outline).

 Step 5: We summarize findings or data from each of the papers relevant to a particular outline 

topic. A second staff reviews the summary of selected findings (number determined by the 

WAM) to ensure that the findings are characterized correctly.

 Step 6: We compile the findings into a narrative that summarize the overall findings of the 

literature review according to the outline.

Data Sets

Other sources of existing data include published databases, data sets that accompany a given study, etc. as 

identified typically through Internet searches, government websites, expert suggestions, or published 

studies identified above. Considerations for identifying potentially relevant existing databases are similar 

to those used to evaluate published literature. The initial review of the source includes assessing whether 

and how the original purpose of the data collection or underlying study may affect usability of the data for 

a different purpose. In addition, the initial review would consider whether other factors such as data year, 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 42

originating agency, collection method, calibration and validation procedure, stringency of the review and 

comment process, etc., fit the data needs agreed to with EPA.

Models

Abt Associates uses models to simulate the effect of the policy or regulation on environmental conditions 

(for environmental assessment or benefits) or affected entities (for cost and economic impacts analyses). 

In selecting models to support analysis of a rule, we review existing models for their applicability to 

simulate the anticipated effects of a regulation – based on the industries, pollutants, affected media, 

processes, ability to specify parameters to represent the policy, and estimate policy-relevant endpoints etc. 

To provide greater transparency of the result and facilitate independent, public review and comment on 

the analyses, we preferably identify models available in the public domain over proprietary models. 

Sources of information about existing models include past EPA rulemaking analyses, list of EPAsupported computational models (e.g., models listed on EPA’s OST modeling portal)

1

, and other Federal 

government agencies such as the U.S. Forest Service and the U.S. Geological Service.

Table 3-2 lists the models Abt Associates uses/used to support current/recent rulemaking analyses. We 

also provide the rationale for selecting these models, based on the criteria discussed in Section 3.1.3.

 

1

The portal (http://www.epa.gov/epahome/models.htm) provides links to models, databases, and tools developed

(including peer review) or supported by EPA offices, labs, and centers that EPA intends for use in regulatory 

analyses to support decision-making.
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 43

Table 3-2: Models used to support regulatory analyses.

Data Source Purpose Criteria for Selecting Resource

SPARROW (Spatially 

Referenced Regression on 

Watershed Attributes)

Environmental Assessment

Regression-based modeling approach. 

Currently used for national modeling of 

the sources, fate and transport of 

nitrogen, phosphorus, and suspended 

sediment in reaches of the coterminous 

United States (E2RF1 resolution).

Federal government source

Existing calibrated and validated model 

applications at the national or regional 

levels

Methodology and existing applications 

documented in peer-reviewed literature

Explicit uncertainty characterization

Run-time conducive to decision-support 

and rulemaking applications

U.S. Geological Survey (USGS) National 

Water-Quality Assessment Program 

(http://water.usgs.gov/nawqa/sparrow/)

SWAT (Soil and Water 

Assessment Tool)

Environmental Assessment

Hydrology, watershed, and water quality 

model. Used to simulate point and nonpoint source pollution in user-defined 

stream networks (variable scale) using 

daily or sub-daily time steps.

Federal government source

Existing calibrated and validated model 

applications in large number of 

watersheds at varying scales

Run-time conducive to decision-support 

and rulemaking applications on selected 

watersheds

Widely used and cited in the literature

Flexible ways to parameterize effects of 

a policy

U.S. Department of Agriculture (USDA)

Agricultural Research Service and Texas 

A&M University

http://swat.tamu.edu/

RSEI (Risk-Screening 

Environmental Indicators)

Environmental Assessment

Analyze factors (toxic chemical releases, 

degree of toxicity and size of exposed 

population) that may result in chronic 

human health risks, based on the Toxic 

Release Inventory. 

Federal government source

National scope

EPA Science Advisory Board-reviewed

Extensive time series (1988-present)

Updated annually

Incorporates EPA standard models (e.g., 

Aermod) and parameters 

Extensively documented

Environmental Protection Agency (EPA)

Office of Pollution Prevention and Toxics

http://www.epa.gov/oppt/rsei/pubs/basic_

information.html

HAWQS (Hydrologic and 

Water Quality System)

Environmental Assessment

Platform to set up and implement SWAT 

models (see above) using pre-populated 

national data sets

Federal government source http://ars.usda.gov/research/projects/proj

ects.htm?accn_no=410569
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 44

Table 3-2: Models used to support regulatory analyses.

Data Source Purpose Criteria for Selecting Resource

IPM (Integrated Planning 

Model)

Environmental Assessment and Economic 

Impacts

Estimate impacts of policies on the 

electricity market, electricity generation, 

air emissions, etc.

Federal government source 

Multi-regional, dynamic, and 

deterministic linear programming

EPA 

http://www.epa.gov/powersectormodeling

/

BenMAP (Environmental 

Benefits Mapping and 

Analysis Program)

Benefits

Estimate the health impacts and economic 

benefits of improvements in air quality

Federal government source

SAB and peer reviewed

Used in numerous studies published in 

peer reviewed journals (e.g., Cohan et al. 

2007; Winebrake et al. 2009; Corbett et 

al. 2007)

EPA Office of Air and Radiation

http://www.epa.gov/air/benmap/

i-Tree Vue Benefits

Estimate benefits of changes to urban 

forest

Federal government source

Peer reviewed

Cited in literature (e.g., Nowak et al. 

2006a; Nowak et al. 2006b)

http://www.itreetools.org/

IEUBK (Integrated Exposure 

Uptake Biokinetic)

Benefits

Estimate blood lead levels in children

Federal government source

Use in rulemaking

Independent validation and verification 

and a limited peer review of the software 

and documentation

Allows for rapid calculations and 

recalculations of an extremely complex 

set of equations

EPA Superfund Office

http://www.epa.gov/superfund/lead/ieubk

faq.htm

Leggett model Benefits

Estimate blood lead levels in adults

State government source (also used by 

EPA)

Use in rulemaking and occupational 

health standards development

Detailed modeling of lead absorption, 

distribution, metabolism, and excretion 

for 21 body tissue compartments on a 

daily time step

Able to address time-dependent 

conditions

Leggett (1993) and 

California Environmental Protection 

Agency, Office of Environmental Health 

Hazard Assessment (Cal OEHHA; 2013). 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 45

3.1.3 Criteria for Selecting Data Sources

Criteria for selecting data sources for this program include the relevance of the source to the data relative 

to the needs, the reliability of the source, and the content. Below, we discuss considerations we apply in 

evaluating potential data sources, focusing on the criteria for published data sets, models, and studies to 

support benefits transfer.

Note that we assume that the data provided by EPA or its other contractors specifically to support the 

rulemaking analysis will have been developed under separate QAPPs that reflect the purpose for the 

analysis; thus, we will accept the data without further review. This section therefore applies to data not 

specifically generated for use in the rulemaking analysis.

Data Sets

In selecting data to support rulemaking analyses, Abt Associates prioritizes data from government sources 

that have been subject to extensive reviews, are widely used and scrutinized, and are known to be of high 

quality. We also preferably select national data sets (e.g., data published by federal government agencies), 

developed using consistent methodology and reporting standards across states, to match the use of the 

data to support analyses of rules that would apply nationwide. For example, EPA and the USGS provide 

water resources data that are national in scope and collected using consistent methods, often with the 

express intent of making the data available to researchers and decision makers. The data are typically 

subject to quality review before publication. Databases published by the U.S. Department of Commerce 

(Census Bureau, Bureau of Economic Analysis), Department of Energy, and other federal agencies 

similarly undergo extensive review before publication. 

Thus, most of the data sets identified Table 3-1 are national in scope while providing varying levels of 

geographic specificity (e.g., state, county, metropolitan area, facility, hydrologic unit), and are therefore 

appropriate for use in national-level analyses. Some data are appropriate for general use across analyses 

(e.g., economic indices or hydrologic network) while others are specific to certain industries affected by 

EPA actions (e.g., electric power sector, construction sector), or specific locations (e.g., state-level 

databases, watershed models). See Table 3-1 for details.

As appropriate, Abt Associates utilizes data produced by comparable state-level agencies as best available

when the data are more detailed than federal government sources or when the data fit the scope of the 

analysis (e.g., developing a case study at a selected location). For example, state governmental agencies 

compile databases of water quality monitoring data for water quality assessment and CWA Section 303(d) 

listing, and sometimes track implementation of pollution control measures for nonpoint sources. We used 

detailed water quality assessment data published by the State of Florida in evaluating the impacts of 

setting alternate numeric nutrient criteria for Florida waters. Such data were more detailed than data 

available from EPA and national-level sources.

For data from all other sources (e.g., peer reviewed journal, government-funded study, trade association 

study), Abt Associates will select the sources through evaluation of data quality and project needs, in 

consultation with the EPA WAM. This evaluation will include consideration of EAD’s needs under the 

Data Quality Act regarding objectivity, utility, and integrity, as described in Section 2.4. We anticipate 

that WA-specific data needs and possible sources will be covered in an SQAPP. The EPA WAM and Abt 

Associates project team will meet to discuss the identified existing data sources for each project. Through 

discussion, the EPA WAM will determine whether sufficient relevant and reliable data sources exist, and 

whether those sources contain sufficient data for the intended use. We will discuss limitations of the data 

set with the EPA WAM and in method memos, including the timing of the data relative to the analysis 

year, level of resolution in terms of spatial scales or industry categories, important gaps (e.g., cover only a 

subset of conditions to be modeled), and apparent biases.
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 46

If data gaps remain, we may identify additional existing data sources, collect primary data, and/or develop 

assumptions or a method to address the data gap (see Section 3.1.5).

Models

Considerations for identifying potentially relevant models are similar to those for evaluating published 

literature and databases. These considerations include assessing whether the purpose, assumptions, 

methods, and data requirements of the model fit the information needs of the analysis, based on review of

the documentation, papers on previous model applications in the published literature, and the 

development status (i.e., whether the model has been peer reviewed). Thus, for models that characterize 

water quality, we consider the types of water bodies and pollutants simulated, the data requirements, and 

reported validity of model estimates in existing applications, to determine whether the model may be used 

to support the environmental assessment and in what application (e.g., national analysis, case study at a 

specific location). We also consider how the policy may be specified in the model, notably whether the 

model provides parameters that may be adjusted to mimic the effect of the policy. To analyze the effects 

of stormwater policies, for example, the selected model needs to be able to mimic the effects of structural 

and non-structural best management practices that may be used to comply with more stringent stormwater 

requirements.

To use existing model applications for a different purpose (e.g., using watershed models previously 

developed to evaluate the effects of climate change to estimate changes in average conditions resulting 

from different stormwater controls in the same watersheds), we also review documentation on the model 

development process, and calibration and validation results, to evaluate how the specific model 

application may affect use for evaluating the regulatory scenarios.

Benefits Studies to Support Benefits Transfer

Benefit transfer can rely on benefit value or benefit function transfer (BFT) as a source of nonmarket 

welfare estimates, with benefit functions drawn from individual primary studies in the literature. The

availability of benefit functions drawn from study sites of sufficient similarity to the policy sites in 

question is likely to be very limited, particularly for application in national-level benefit analyses. In cases 

for which a closely matching study site or policy function is not available, a parameterized meta-analytic 

benefit function, estimated using a meta-regression model of many primary study results, can provide the 

analyst with a viable alternative. We determine the optimal scope of the metadata based on the exact 

definition of the dependent variable in the meta-regression model, which, in turn, defines the set of source 

studies that can be considered for inclusion in the metadata. The primary tradeoff is often between 

maintaining close similarity among dependent variables versus including additional information (i.e., 

observations) in the metadata. 

For example, in compiling resource valuation literature to support meta-analysis of surface water and 

open space valuation studies, we use the following screening criteria for identifying candidate studies: 

 Specific amenity being valued:

 Surface water quality – The amenity being valued is limited to those in which a water quality 

change affects ecological services provided by waterbodies, including aquatic life support, 

recreational activities (such as fishing, boating, and swimming), and nonuse value. For 

example, a study that estimates WTP for recreational fishing improvements due to fish 

stocking would not be included in the metadata. However, a study that estimates WTP for 

recreational fishing improvements due to nutrient reductions would be included in the 

metadata, presuming it satisfies all other criteria. 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 47

 Open space (stormwater best management practices (BMPs)) – Very few studies directly 

address values related specifically to stormwater BMPs, some do not estimate dollar values, 

and most use proxy measures, such as open space, to evaluate effects of these practices. To 

develop a valuation function that could be used to value an increase in vegetation and open 

space due to implementation of stormwater management BMPs, we compared features of 

stormwater BMPs (including site redesign to reduce impervious cover) to various types of 

open spaces considered in the valuation literature and selected studies of open space similar 

to open spaces provided by development site redesign to reduce impervious cover (e.g., 

increased vegetation; trees and forests; and water-related, but not “open water”). 

 Values estimated – Following the requirement for welfare consistency, the metadata will not 

combine studies representing theoretically distinct welfare measures (i.e., stated and revealed 

data). 

 Surface water quality – We focus on assembling information from studies using stated 

preference approaches because such studies elicit total household WTP (including use and 

nonuse values). Revealed preference studies allow the estimation of use values only. 

 Open space (stormwater BMPs) –We limit selected studies those that used hedonic valuation 

techniques to measure a percent change in home value as most relevant studies for estimating 

changes in property values.

 Amenity detail – As described by Johnston et al. (2005), “the study must provide sufficient 

information regarding resource, context, and study attributes to warrant inclusion in the metadata” 

(p.223).

 Ability to convert environmental quality measures to similar terms – Original studies may use 

different metrics to measure environmental quality, to standardize metadata we will convert the 

relevant measures of environmental quality change into a single common metric. Some studies 

may not be included in the final data set because the environmental quality measure could not be 

standardized due to lack of information.

 Surface water quality - We plan to use the water quality index (WQI) as the metric for water 

quality improvements within the meta-analysis, consistent with the existing model. The WQI 

scale is used to link specific pollutant levels to suitability for human uses. Some studies can 

be more easily incorporated if they use a standard WQI or water quality ladder (WQL) to 

measure water quality changes. Other studies may rely on ordinal ranking or verbal 

description to measure water quality. In these cases, provided available information in the 

study is sufficient, we will use professional judgment to map these rankings and descriptions 

to analogous WQI levels. 

 Open space (stormwater BMPs) - Original studies estimated the value of open space 

amenities in terms of a marginal change in one of several different metrics: either a home's 

distance to open space, a home's adjacency to open space, or the proportion of a home's 

neighborhood kept in open space. To support meta-analysis we convert all measures into a 

single common metric-percent change in open space within a radius around the property. 

 Study location – All studies must be conducted in the U.S. or Canada.

 Research methods – Studies must apply research methods that are supported by the literature.

 Duplicative studies – Some studies may be released in multiple forms, such as working papers, 

conference papers, journal articles, and book chapters. Articles will be screened to remove 

duplicative analysis and analysis that were subsequently revised and published. Multiple values 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 48

can be included for a study, however, if they address a different component of the study data or 

estimate values for alternative resource changes.

To document metadata we follow reporting guidelines outlined by Stanley et al. (2013) for metaregression analyses in economics to record, among other information: (a) the exact databases and other 

sources searched, (b) the precise combination of keywords, and (c) date completed. In addition, Abt 

Associates will compile the following study information in electronic format:

 Full study citation

 Study location

 Sample data and description (e.g., size, response rate, income)

 Resource characteristics (e.g., affected waterbody or land type, recreational uses, baseline quality)

 Description of environmental quality change, including geographic scale and affected recreational 

uses (e.g., water quality change from fishable to boatable or percent increase in open space)

 Quantitative measure of environmental quality change 

 Original WTP values, that we update to current year dollars using the Consumer Price Index,

 WTP estimation characteristics (i.e., parametric versus non-parametric, inclusion of protest bids 

and outlier bids, WTP description), if relevant. 

3.1.4 Data Value Selection Approach

As part of the data source selection and data gap evaluation described above, Abt Associates will work 

with the EPA WAM to select data values (or data sets) from the potential sources identified. We will 

discuss the selection criteria with the EPA WAM (e.g., age of the data, coverage and resolution, prior use 

in rulemaking, use in other studies), the relative importance of the criteria in the decision process, and 

prepare a matrix summarizing how each data source scores according to the metrics. We will only use 

metrics that can be objectively evaluated to ensure transparency in data selection. From the initial review 

of data content, we may find that:

 One source is the most relevant and reliable. If we find only one relevant and reliable source, 

then we will document the selection of this source, subject to EPA WAM approval.

 More than one source offers the data element(s) of interest either individually or when combined.

If we find several relevant and reliable sources we will examine the range of values and 

recommend an approach, subject to the EPA WAM’s approval. For example, we may recommend 

using the average of values from the multiple acceptable sources or using data from a preferred 

source that best fit the data needs (e.g., most recent, most representative) or other criterion 

identified by the EPA WAM (e.g., most cited; previously used by EPA, for example, for 

comparative purposes). Abt Associates will investigate the range of data values found, whether 

the data represent normal distributions, or whether the data appear to reflect other distributions. In 

cases of multiple data points that are normally distributed, averaging the data for an arithmetic 

mean is appropriate. In cases of multiple data points that are not normally distributed, then using 

a median of the data may be more appropriate. Finally, when EPA does not require a single value 

for the analysis, presentation of the value range may be most appropriate. Abt Associates will 

work with the EPA WAM in deciding which approach to use for multiple acceptable values, and 

document the process and decision in a memorandum to the file or update to this PQAPP.

 No sources yield the data elements of interest. If none of the sources are relevant and reliable, we 

may conduct further data searches, obtain data through primary data collection (e.g., phone 

contact; primary survey), make assumptions or develop a method to estimate the needed value, or 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 49

not estimate the value and describe potential beneficial effects qualitatively. For example, we 

developed and implemented a stated preference survey to estimate total use and nonuse value of 

aquatic habitat improvements resulting from regulation of cooling water intakes because existing 

economic studies did not provide a good match to the policy context. We used qualitative 

description of secondary benefits such as fishing equipment sales associated with an improvement 

in recreational opportunities resulting from ELGs and other CWA regulations.

In the case of human health risk estimates, we will identify the endpoints with sufficient weight-ofevidence for inclusion in estimating benefits. In the case of carcinogens, we may use unit risk estimates 

provided in the Integrated Risk Information System (IRIS). After consultation with the EPA WAM, we

may conduct supplemental modeling (using BMDS) if addition information is needed beyond what is 

provided in the IRIS documentation. 

We may supplement these values, particularly for noncancer endpoints, with concentration-response 

functions adapted from published epidemiological studies. In this case, for endpoints with sufficient 

weight-of-evidence, we will look for well conducted studies representative of the population under study. 

We will give preference to studies that treat contaminant exposure as a continuous rather than categorical 

variable, and adjust for important confounders.

We will document the data selection process in the analysis documentation, including any noted caveats

or limitations that affect use of the data.

3.1.5 Resolving Data Gaps

Upon completing a review and draft selection of data, we discuss with the EPA WAM any quality issues 

or limitations that could affect anticipated uses of the data to support analyses. We also discuss possible 

approaches for resolving data gaps. Methods for filling in data gaps include, but are not limited to, 

interpolating between two values, imputing missing values using the average or median of known 

observations, using data available at an aggregated level of detail to impute values at a finer level of 

detail, and extrapolating data from a sample of observations to other observations with similar 

characteristics.

For example, if data are available only at a more aggregated level of detail than EPA is seeking for the 

analysis (e.g., Hydrologic Unit Code (HUC)10 versus HUC12 level; 4-digit North American Industry 

Classification System (NAICS) compared to 6-digit NAICS level), we would discuss whether to do the 

analysis at the more aggregated level, or disaggregate data to the finer level of detail using proxy metrics, 

recognizing the uncertainty involved in doing so. In modeling the effects of stormwater controls on water 

quality, for example, we used projected development data at the level of HUC12 basins as input to a 

watershed model that relies on development at a finer level of detail. Mapping the development to 

modeled subbasins required distributing the projected development in proportion to the baseline 

distribution of development across the subbasins.

In addition, as described in Section 3.1.4, it may not be feasible to resolve data gaps. In such cases, we 

fully discuss the data limitation in a section of the report entitled Limitations and Uncertainties, including 

the likely resulting direction of bias, and any information on the potential magnitude of the bias, on the 

final cost, benefit, and economic impact estimates.

3.1.6 Data Gathering Documentation and Records

Data gathering documentation and records depend in part on the type of existing data.
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 50

Published Literature 

Abt Associates collects and organizes literature using reference management software (EndNote) when 

the literature review task involves more than a few articles, or covers topics that are germane to several 

project activities and thus may become a data source for use in later work assignments. In particular, we 

use reference management software to maintain an up-to-date repository of the literature on topics 

germane to the types of analyses or methods often conducted for this program, such as valuation methods 

for different benefit categories, dose-response functions, water quality indices, and statistical procedures.

Using reference software allows for efficient compilation of full bibliographical information, facilitates 

citing of sources in final reports, ensures consistency in the way metadata information is recorded, and 

minimizes opportunities for transcription errors when citing literature.

Abt Associates’ Research Assistants, Associate Analysts, and Analysts are typically responsible for 

identifying and screening the relevant literature, and for summarizing the data or findings relevant to the 

research task. The team accomplishes this by noting relevant aspects of the study specific to each research 

question. For example, the research notes may include the period of the study, type of waterbody, spatial 

or temporal resolution or scope of the study, pollutants evaluated, study method, conclusions, and 

important limitations or uncertainties that may affect the use of the study results in other analyses (see 

Section 3.1.2 ). As described in Section 3.1.3, for meta-analyses, we follow reporting guidelines outlined 

by Stanley et al. (2013) for meta-regression analyses in economics to record, among other information: 

(a) the exact databases and other sources searched, (b) the precise combination of keywords, and (c) date 

completed.

The Abt Associates WAM, or a designee, is responsible for verifying the accuracy of the data extracted 

from the original source. We may accomplish this verification by independently reviewing a selected 

number of papers and comparing the literature review notes, or by interviewing the reviewer to gauge 

understanding of the fine points of the study and applicability to the target analysis for which the 

literature review is being conducted.

We often transcribe values from published articles through electronic cut and paste, or manually. As a 

result, transcription errors may occur. We verify the accuracy of values extracted from the literature by 

directing a second individual, designated by the Abt Associates WAM, to confirm that (1) the value used 

in a spreadsheet or cited in the document corresponds to the value in the original source, (2) the units (and 

unit conversions) are correct, and (3) any confidence interval or uncertainty bound provided in the 

original source, are included. We perform this transcription QA as part of the spreadsheet or document 

QA review discussed below. 

Electronic Data Sets

Abt Associates collects and organizes electronic data sets in project folders organized by task or analysis 

components. We retain copies of all data in the original “as received” format, along with any associated 

metadata, related publications, source Web page, email message, and other source information. Table 3-1

provides the criteria for selecting the electronic data sets commonly used under this contract. Section 4.1.4

describes our procedures for databases and data set transfer and maintenance.

Telephone and Email Data Collection

Abt Associates documents conversations used as “data sources,” and generates corresponding electronic 

files in the project folder. Abt Associates maintains the following documentation procedures when 

collecting data that may be used to support analyses conducted under this project via telephone and email:

 Document telephone contacts using a Telephone Contact Report depicted in Appendix D.
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 51

 Save email contacts as portable document format (PDF) files, and add to the electronic project 

file.

The Abt Associates WAM and EPA WAM will determine which data materials gathered through 

telephone or email conversation during the project need to be included in any record.

Hardcopy Documents

In the event that we receive data as hardcopy documents, Abt Associates scans the relevant parts of the 

document and works with the data in the same way as described above for published literature. The 

project team retains a hardcopy of the document in physical folders and provides the document to the 

EPA WAM at the end of the work assignment.

3.2 Data Management

This section describes the procedures and best practices that Abt Associates employs to manage data 

under the individual projects.

3.2.1 Standardization of Data Elements

In developing analytic frameworks, Abt Associates defines standard categories that will be used 

throughout the analysis to ensure consistency in concepts and alignment of data with respect to temporal, 

spatial, industry sector, and other dimensions. Establishing and using standard categories is a best practice 

that ensures uniform reporting of data across the various parts of the analysis.

For each needed adjustment, we document the adjustment factor or crosswalk used to convert the 

secondary data into the framework. For transparency and reproducibility, we retain in our files the 

original value provided in the data source, along with the adjusted values.

Table 3-3 summarizes standard data elements for analyses conducted for EAD.

Table 3-3: Standard data definitions for EAD analyses.

Standardization 

category

Standardization 

basis

Implications for handling of secondary data

Temporal Analysis year If needed, we bring data forward to the analysis year. For costs, we use 

CCI, ECI or other appropriate indices or adjustment factors, depending 

on the type of cost (see Table 3-1).

Spatial State or counties We code data using the state postal abbreviations or county FIPS as 

identifiers

ComID We code waterbodies using their ComID (if doing the analysis at the 

NHD level), reach ID (if using E2RF1), or other unique identifier (e.g., 

Florida DEP WBIDs).

HUC We code watersheds using the HUC system. We identify the appropriate 

HUC(s) in ArcGIS as needed. 

NERC region We code electric power grid regions using EIA’s NERC regions 

definitions. 

Economic Sector NAICS If needed, we categorize data provided for facilities or economic 

activities using the appropriate NAICS sector(s).

Chemical CAS RN For analyses involving calculations across chemicals, we use the CAS 

RN to identify unique chemicals.

Electric power 

generator

EIA ID We identify the EIA ID for electric power generators, operators, etc. to 

facilitate cross-references to DOE data on the operational characteristics 

of the plants.
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 52

Table 3-3: Standard data definitions for EAD analyses.

Standardization 

category

Standardization 

basis

Implications for handling of secondary data

CAS RN = Chemical Abstract Service registry number

CCI = Construction Cost Index

ComID = Integer value that uniquely identifies the occurrence of each feature in the NHD

DEP = Department of Environmental Protection

DOE = Department of Energy

EIA = Energy Information Administration

ECI = Employment Cost Index

HUC = Hydrologic Unit Code

NERC = North American Electric Reliability Corporation

NHD = National Hydrography Dataset

WBID = Waterbody Identification

During the development of the framework, we also define the units we will use to conduct the 

calculations, summarize the data, and present the results (e.g., land area used as hectares in the watershed 

model but presented in acres; loadings calculated in kilograms but presented in tons). As needed, we 

convert data from their existing units to the calculation or reporting units using universal conversion 

factors (e.g., obtained from Excel or Google built-in unit conversion tools, or onlineconversion.com). 

3.2.2 Data Entry

Abt Associates Research Assistants, Associate Analysts, and Analysts typically are responsible for 

collecting existing data and developing the associated databases and spreadsheets. Prior to using the data 

in any analysis, the Abt Associates WAM or a designee reviews the data for accuracy and 

appropriateness. For data entry, we check and document that the data and other information accurately 

represent the original source by verifying 100% of manually entered data points.

3.2.3 Merging or Uploading Electronic Data from Existing Sources

The steps for ensuring the quality of the data extracted, merged, or compiled from electronic sources 

depend on how the data are to be incorporated in the analytic frameworks.

If practicable, we input data directly into the analytic framework from source files, with the analysis 

framework or model providing a unique link to any shared information – i.e., the analysis relies on the 

same data stored in a unique location to avoid potential inconsistencies.

After extracting or merging data from existing sources, we verify the following:

 The extraction program did not generate any errors (e.g., if importing into an Access database, the 

software program did not warn of values that could not be converted into any new data format);

 The number of records extracted match those in the original source (i.e., the program used to 

extract or store the data did not truncate the data set);

 The row and/or column totals match those in the original source or in a summary report that 

describes the original source;

 All columns have valid values, as verified by visual inspection (e.g., numeric values rather than 

blanks, or no negative numbers, based on knowledge of the file contents); and
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 53

 Data in selected records match those in the original source exactly, as verified by manually 

comparing selected records (5-10 records in the case of the process is automated and record 

formats are consistent throughout the data set).

These procedures enable us to confirm that the extraction program did not generate any errors. We 

describe additional data manipulations to ensure that the extracted data use standard categories, and are 

consistent with the analytic framework in Sections 3.2.1 and 3.3.2, respectively. 

In addition to the merged data, we retain a copy of all existing data used in analyses as received from the 

original source on a particular date, documenting all changes to the original values, along with the reason 

for any adjustments (e.g., unit conversion, standardization to analysis year). Abt Associates staff cite the 

date and origin of the data in all spreadsheets, databases, and documents, and provide hyperlinks to online 

sources for easy reference. We use this metadata during our quality assurance review of the framework to 

assess data sources and, at later stages of the project (particularly when analyses may take place over 

several years), to identify input data sets that may need to be updated based on the publication of more 

recent data.

3.2.4 Data Review

Sections 3.2.2 and 3.2.3 describe Abt Associates’ procedures for verifying that the data values we enter 

manually, or download electronically, match the original source. After data verification, we review the 

data to identify any remaining data gaps, zero or missing values, values with QA issues, or outliers (e.g., 

values outside the expected range, negative values, or values orders of magnitude higher than reasonable). 

We correct such values when the data exist to do so, or remove them from the analysis, documenting in 

the file the reason for the change or exclusion.

3.2.5 Data Storage and Manipulation

Abt Associates compiles existing data in spreadsheets (e.g., Microsoft Excel), relational databases (e.g., 

Microsoft Access, MySQL), reference management tools (e.g., EndNotes), and other software tools (e.g., 

ArcGIS, R) to facilitate their use in analyses. Abt Associates has dedicated information technology (IT) 

staff responsible for the installation and maintenance of all hardware and software infrastructure, 

including automated daily incremental backup of all data stored on networked resources. The Abt 

Associates WAM will work with the IT staff to ensure that data storage procedures are supported by the 

IT infrastructure. At the completion of this contract, Abt Associates will archive the data (including 

analysis results) so as to preserve these data with no further modifications. 

Abt Associates stores and handles electronic files containing CBI data in accordance with our CBI 

Security Plan for this contract, which ensures that only staff with need-to-know have access to the files

and any derived analyses that are also CBI. 

Abt Associates employs best practices in the use of software to store and manipulate data, such as using 

queries and lookups in Access and Excel to extract relevant portions of datasets, and including 

intermediate results to facilitate review. The PQA Advisor for an individual work assignment, or his or 

her designee, reviews all calculations within spreadsheets and databases by hand to verify that formulas 

are correct and producing the intended result, and hence the data in the columns are correct.

For example, spreadsheet best practices minimize the risk of errors through structured design and 

consistency; such practices also ensure transparency and efficiency in update time and costs. As an 

example, the Spreadsheet Standards Review Board (http://www.ssrb.org/) develops and publishes 

spreadsheet best practices that cover 16 areas of spreadsheet model development. These include general 

concepts; workbook structure; sheet structure; formats and styles; assumption entry interfaces; sensitivity 

analysis; outputs and presentations; calculation formulae; naming principles; time series analysis; checks; 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 54

printing and viewing; multiple workbooks; security and protection; Visual Basic programming; and 

miscellaneous. These guidelines are available to all staff for implementation as applicable, although not 

all areas may be applicable to any one project. Therefore, Abt Associates staff tailor implementation to 

the complexity of the existing data and forthcoming analysis, while maintaining the framework for 

revising and adding data and complexity at minimal cost. 

3.3 Data Quality Assessment and Use

After identifying and selecting existing data sources, Abt Associates and the EPA WAM review the data 

to assess usability for the analysis. As described in Section 3.3.1, we compare our findings regarding data 

accuracy, reliability (e.g., presence of apparent outliers or unresolved questionable values), and 

representativeness (e.g., completeness, resolution, variability), against the objectives. We document the 

limitations and gaps identified during our internal review in a memorandum to the EPA WAM, along 

with proposed approaches for addressing these limitations, when possible. We then decide with the EPA 

WAM on approaches to be used to address the limitations. These approaches may include moving ahead 

with the analysis using the existing data and detailing the limitations, along with implications for 

interpreting the results, in the final deliverable describing the analysis; collecting additional data targeted 

at specific gaps (e.g., collecting additional data for a selected region for which the national dataset is 

lacking or has questionable values); or pursuing primary data collection. 

3.3.1 Data Quality Verification and Data Quality Reporting

Abt Associates staff will evaluate existing data against objectives of: (1) accuracy, (2) reliability, and (3) 

representativeness, considering the data source, available documentation, and content. 

Abt Associates assumes that data and information contained in state and federal databases and reports are 

accurate and reliable to the extent specified in the quality assurance and control documentation 

accompanying the source. For these sources, we assess and identify any known limitations or caveats, and 

summarize the data to confirm the scope and identify possible outliers (e.g., data that are not consistent 

with the body of information collected) and other data issues that may affect use of the data. 

For other data sources (e.g., peer reviewed literature, trade publications), Abt Associates reviews the 

documentation and content to identify potential for biases or other data quality issues (e.g., data are based 

on a limited sample). We consider the quality of the documentation, clarity of presentation, discussion of 

uncertainty, and scientific merit of the methods and assumptions to assess whether the data source is of 

sufficient quality for use in the analysis, and if so, note any limitations. Abt Associates will communicate 

these data quality issues, and discuss the best approach to address the issues, including gathering data 

from several such sources and varying conditions.

Abt Associates will report the outcome of data quality assessment to the EPA WAM in meetings or short 

written deliverables (e.g., memoranda, tables), as needed, and record findings in the project file.

3.3.2 Use/Analysis of Existing Data/Data Operations

Abt Associates uses existing data to describe baseline conditions, and as input to environmental and 

economic models for evaluating the impacts of regulatory options. This section describes how we use the 

data within these frameworks, focusing specifically on data for environmental and economic models. We 

may implement these models in a variety of software, such as spreadsheets, structured programs, and 

relational databases).

We follow specific procedures when conducting complex modeling assignments, such as when running 

watershed or economic impact models to support rulemaking. (These procedures embed documentation 

elements also described later in Section 3.3.3.)
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 55

At the onset of each scenario development, we prepare a table detailing the various elements of the 

scenario specifications: watershed(s) or plant universe, model run years, output parameters, version of the 

model (e.g., date of executable file), analysis baseline (e.g., date input was received), list of input files 

modified since the last model run, changes to default model assumptions, etc. We use this table to record 

metadata for each scenario, and to track progress in completing the model runs.

We implement the steps in the modeling process using algorithms, batch programs (in SAS, R, 

VisualBasic, or other scripting language), and spreadsheets, to prepare model input data, run the model, 

and post-process the outputs automatically, or at a minimum perform several interim steps automatically. 

We generate log files as part of the batch program to audit each step of the process and verify that the 

scenario was specified correctly when reviewing the results. We also build in validation checks at various 

stages in the algorithm or the spreadsheet to flag any conditions that could affect the analysis. For 

example, we build in QA by including code in the algorithm to verify that interim or final calculations fall 

within a valid range (e.g., BMPs have calculated drawdown times of at least 48 hours, all values are 

positive) and to generate an error when the conditions are violated.

When developing or modifying an algorithm or batch program, we verify that the program performs the 

calculations correctly by reproducing the calculations offline (e.g., in Access or Excel) on a sample data 

set and/or verifying known equalities or values in internal validation calculations within the program. The 

approach depends on the type of algorithm being developed or the extent of the modifications. We review 

algorithms and calculations by having a second person not directly involved in developing the 

algorithm/program review the code to ensure its accuracy. The Abt Associates WAM is responsible for 

determining the extent and methodology for the review. Thus, we may review major algorithm 

development and modifications by doing the calculations offline. For relatively minor changes, such as 

changing assumptions or inserting conditional statements within a largely unchanged algorithm (e.g., 

adding a conditional clause to subdivide treatment of conditions using If-Then statements), the Abt 

Associates WAM may decide to approve the changes after conducting a code review, without requiring 

reproduction of the calculations offline.

We conduct first-level quality control of model runs by verifying that all steps involved in preparation of 

the input files and processing of the model outputs were completed correctly. In conducting this review, 

we consult the log file generated by the batch programs and any additional log files generated by the 

model. 

We process the raw model outputs using an algorithm and batch program customized to the particular 

model outputs (e.g., SWAT, cost impact model). For example, we process model outputs to generate the 

desired statistics (e.g., average daily water quality index score for each analyzed reach, number of entities 

exceeding cost-to-revenue thresholds) along with intermediate calculations and summary statistics to 

allow auditing of the results. Thus, analysis of daily reach-level outputs that we obtained from the SWAT 

model includes minimum, maximum, average, median, and quartile statistics for each water quality 

parameter and flow to allow comparison of the distribution of daily values and not only indicators of 

central tendencies.

We review these outputs to assess whether the results are within the expected range. For models that 

generate a large number of observations (daily values on all reaches analyzed over a 30-year period), we 

look at indications that the distribution of values is reasonable by looking at the minimum and maximum 

values. Once the data have passed this initial check, we look at the changes across scenarios (or relative to 

different runs of the same scenario) to assess whether they are in the anticipated direction. 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 56

We save summary results, raw model outputs, log files, any model input files modified in running the 

particular scenario, and the algorithm used in analyzing the scenario in a separate folder. Keeping all files 

together ensures that we can reproduce the results at a later date if needed. 

3.3.3 Methodology Documentation and Conceptual Review

In developing methods for data analyses, Abt Associates documents proposed approaches and options in 

methods memorandums. The Abt Associates WAM may involve the Project Manager in developing the 

methods, and involves the PQA Advisor in review of the memorandums. The Abt Associates WAM 

provides the memorandums to the EPA WAM, who reviews and comments, and involves the EAD 

Branch Chief in selecting the approach. Abt Associates typically prepares method memoranda in a format 

that facilitates insertion into the final report or technical support document for the regulatory effort.

The PQA Advisor or his or her designee reviews the spreadsheets, models, and programs against the 

methods memoranda when performing and documenting quality control (see Appendix A for the checklist

template we will use for documenting PQA Advisor reviews under this contract). The WAM adjusts the 

checklists as needed based on the complexity of the analysis to account for the flow of data from the 

source data to final deliverables, including location of the spreadsheets, databases, and computer 

programs; staff who ran the analyses; location of the input data; version of the data; and which iteration of 

analytic program or frameworks resulted in the numbers that we report.

We also use the following procedures, as appropriate to the analysis, to document the methods and flow 

of the work:

 Document the process (flow) of the data work in the appropriate task-level directory, including:

 Name and location of input files

 Name and location of program

 Name and location of output files or tables

 Date the program or process was run

 Document the order of steps within spreadsheets, programs, or separate file within the appropriate 

task-level directory.

 Catalog incoming and outgoing (sent outside the project) data files, including name, source, and 

date

 Maintain crosswalks between report tables and origin (e.g. Excel workbook)

 Include documentation within analytical framework or program code, including:

 General information at the top of every framework or program, such as name and path of the 

program, the name of the project, the name of the programmer, names of input and output 

files, and the purpose of the program

 Titles and/or footnotes in any output

 Sufficient comments to provide a‘roadmap’ to the analysis flow

 Track processing steps and program runs

 Store all outputs of analysis runs

 Archive programs and all project documentation.

The Abt Associates WAM, working with the project-level PQA Advisor, also reviews that the analyses 

implement the methods described in the method memoranda, including any test or pilot runs to verify that 

the methods effectively produce the analytical result agreed to with the EPA WAM. The Abt Associates 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 57

project team discusses any issues regarding the conceptual review with EPA WAM, and documents any 

agreed-upon changes to the methods in memoranda, as needed, and monthly progress reports. 

3.3.4 Technical Review of the Data Analysis

This section describes Abt Associates’ process for ensuring that project staff are properly performing data 

analysis to support the project.

Section 3.2.5 describes the electronic and manual spreadsheet best practices and quality control checks 

that Abt Associates staff will use to verify the accuracy and usability of data to meet the project 

objectives. Abt Associates keeps record of spreadsheet reviews, including the name of the review and the 

date of review completion, within the spreadsheet coversheet. Abt Associates analysts will also perform 

analyses of existing data within spreadsheet workbooks consistent with the best practices described 

above. As such, under best practices, existing data will be separated from assumptions, outputs, and 

presentations, which facilitates data verification. Appendix A shows the checks that the PQA Advisor or 

designee, will be responsible for completing related to assumptions, summary statistics, 

models/algorithms; formulas; outputs; and presentations that we compile and develop within 

spreadsheets.

The WAM will add WA-specific detail to the QA template, as needed, to facilitate PQA Advisor review.

Any time that we make material changes to the models or calculations, the Abt Associates WAM or 

designee reviews the inputs, calculations (including correct reading/writing of the inputs and outputs), and 

outputs to ensure that we have processed the data correctly. For models for which calculations have 

already been verified, the review focuses on the model inputs to ensure that they are accurate, and on the 

outputs to ensure that they are complete, consistent, and properly described. 

3.3.5 Final Verification of Data Analysis and Reconciliation with User Requirements

This PQAPP identifies the level of review required for project deliverables, and who will conduct the 

review. Abt Associates conducts this review using the quality assessment tools described in Section 3.3.1

of this PQAPP.

Upon discovering issues that affect any delivered work product, the Abt Associates WAM immediately 

informs the EPA WAM of the change and of any resulting modifications to QA procedures to prevent 

recurrence of the problem. If needed, Abt Associates revises the PQAPP to reflect changes in the 

procedures.

The Abt Associates WAM also immediately informs the Quality Assurance Manager, as well as the PQA

Advisor and Project Manager, of material errors that passed the established QA process, to identify and 

evaluate mitigating measures to prevent recurrence and resolve the issue.

Abt Associates includes a summary of QA-related activities as part of its monthly progress reports, in a 

section titled: “QA Activities during Reporting Month.” The summary describes steps taken to assess the 

quality of data used in the analysis, verify that the data, analysis frameworks, and models used to support 

programmatic activities meet the data quality objectives and are appropriate, and verify the accuracy of 

deliverables to EPA. For example, we will indicate when files have been reviewed, and document any 

changes to the analysis or draft/preliminary results. Preparing the monthly progress reports prompts the 

Abt Associates WAMs to identify and report on the QA activities conducted during the month, and

consider the QA activities that should be performed in the next reporting period.

For example, QA activities reported under WA 05, Steam Electric ELG, include:

 Compared most recent Energy Information Administration (EIA) data (for 2012) to data used in 

the existing analytic framework (EIA 2009 and Steam Electric Survey) to identify possible issues 
Existing Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 58

before incorporating the new data (e.g., change in ownership, retirements). Abt Associates will 

describe and review these issues with the EPA WAM before bringing new data into the 

framework.

QA activities reported under WA 03, Post Construction Economic Analysis, include:

 Performed additional QA/QC of all models and input /output data by using cross-walks across 

different parts of the analysis to ensure consistency, and by using spot check/manual calculations 

to validate model results (e.g., compare maximum BMP storage volume provided in model output 

file to value calculated manually using input specifications for BMP capacity and development 

area managed (storage depth ×total development area ×fraction of development managed by the 

BMP), check mass balance on the BMP by comparing total water inflows to the total water 

outflows and changes in water storage).

 Verified that exploratory analyses of SWAT model outputs were done correctly (e.g., data are 

extracted from the correct file(s), data represent the desired metric, necessary unit conversions are 

implemented correctly, and summary statistics are properly calculated), the analysis approach 

provides insight relevant to each study question, and additional questions emerging from the 

analysis are identified and either resolved or flagged for EPA attention.

 Verified that extrapolation approaches are implemented correctly (correct data extraction, 

appropriate model specifications), and that the results are appropriately described and qualified.

 Performed QA/QC of small-scale model inputs to ensure that land use change and policy 

scenarios are implemented correctly and are consistent with inputs for larger scale model (e.g., 

land use changes reflect projected development, differences between the amount of development 

in the small vs. large-scale models are consistent with the relative watershed vs. subbasin areas).

 Pulled data for selected watersheds and model components, and compared results to engineering 

outputs and across locations and scenarios to understand conditions and processes that contribute 

to increased loads and/or concentrations simulated in SWAT. Corrected equation used to reset 

phosphorus limits in SWAT when effluent concentrations are specified for a given BMP; under 

certain circumstances, the calculations could result in division by zero. Reran SWAT models. 

Change did not materially affect results obtained previously.

The Project Manager for this contract holds bi-weekly meetings with all Abt Associates WAMs. These 

meetings provide an opportunity to exchange ideas on best practices, QA issues, etc. that affect 

programmatic activities under this contract. Additionally, the Project Manager and senior managers meet 

once per quarter with the Division Vice President to discuss contract status, including any QA issues that 

may affect performance under the contract, and resolve any issues by taking appropriate actions at the 

company level.

Abt Associates includes a QA section in final project deliverables (e.g., major rulemaking support 

documents, final project reports) that addresses the data collection, production, analyses, and use.
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 59

4. Primary Data

This section describes Abt Associates’ procedures for data gathering, management, and quality assurance 

and use pertaining to primary data; primary data are new data generated specifically for use in the 

program.

Abt Associates will support EPA in developing, administering, and 

analyzing results of electronic questionnaires to gather data from

facilities in specified point source or industry categories to support 

ELGs. This support corresponds to steps 4 and 7 in the ELG 

development flowchart of Figure 2-2. The EPA WAM will define the industry, source category, or other 

entity or population subject to a data collection effort.

Figure 4-1 illustrates the general process for developing and implementing industry surveys. We discuss 

considerations for data gathering and data quality assessment and use that are pertinent to the conduct of 

industry surveys in Sections 4.1 and 4.2. 

Figure 4-1: General Process for Developing and Implementing Industry Survey.

Data Needs Draft Questionnaire Final Questionnaire Pre-Test Obtain sample Develop ICR Supporting Statement Analysis requirements Data gaps Define sample frame Needed resolution Format

Follow-up with respondents Provide support Tracking / Receipt

Compile Individual Responses in Draft Database Verify data completeness and accuracy Put survey in the field (Mail to respondents) Design/Setup Phase Implementation Phase Reporting Phase

Survey results report and Final Database Document database and limitations

Assess data gaps and quality

Follow up with respondents to confirm/ revise data Fill-in data gaps Produce Final Database

In addition, EPA may also direct Abt Associates to develop and administer smaller survey efforts to

collect specific primary data from a subset of entities2or may direct Abt Associates to collect primary 

data from the public to support monetization of benefits (e.g., stated preference surveys). The sections 

below focus on industry surveys and small data collection efforts. We describe the general process for 

conducting interviews to collect primary data in Section 4.3. We will address additional procedures for 

conducting stated-preference surveys, if requested by EPA, in a work-assignment specific SQAPP. 

 

2 A survey of nine or fewer entities for the same information does not require an Information Collection Request 

(ICR).

Primary Data

 Industry surveys

 Industry contacts and 

site visits
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 60

Note that the statement of work under this contract does not involve laboratory or field measurement; as 

such, this PQAPP does not address elements specific to laboratory analytical methods, equipment, and 

supplies.

4.1 Data Gathering

This section describes data gathering elements for primary data collection that differ from those for 

existing data collection. 

4.1.1 Sampling Process (Experimental) Design

The experimental design and process for collecting primary data will depend on the following factors:

 Types of entities and data collection instrument(s). For industry surveys, EPA and Abt Associates 

may perform data collection using one or more survey instruments within the same or consecutive 

phases. EPA may choose to distribute screener and/or detailed questionnaires. EPA uses screener 

questionnaires to identify entities that may be subject to a regulation, and/or to identify a subset 

of entities that should participate in detailed surveys to help reduce the burden of a full survey 

effort. As an example, for the CWA section 316(b) rulemaking, EPA conducted a screener survey 

to identify electric power plants with cooling water intake structures that would likely be subject 

to a 316(b) regulation. EPA subsequently targeted these facilities in its more detailed 

questionnaires to support the rulemaking effort. For the Steam Electric industry survey, EPA did 

not use a screener survey but distributed the questionnaire to all coal- and petroleum coke-fired 

power plants, and to a statistically sampled subset of the rest of the steam electric industry, 

including oil-fired, gas-fired, and nuclear power plants in a single phase. 

 Types of data. EPA generally develops multiple questionnaire parts and sections as needed so that 

questions on a similar topic can be grouped together. EPA may also direct questionnaires at 

specific types of entities (e.g., publicly owned treatment works recipients to assess the impacts of 

pretreatment). As an example, the Steam Electric questionnaire consisted of nine parts (Parts A 

through I) with different parts targeting certain types of power plants or power plant operations. 

Abt Associates assisted EPA with developing a specific section covering financial and economic 

information applicable to all types of plants. 

 Data collection instrument format(s). EPA has used both paper (hardcopy) and electronic 

questionnaires in industry surveys used for past ELG development efforts. Because of the 

efficiency of electronic questionnaires, we anticipate that EPA may conduct future industry 

surveys under Contract EP-C-13-039 using electronic questionnaires which will either be mailed 

to respondents or delivered over the Internet, if possible given data security requirements (e.g., 

ensure CBI protection). Accordingly, we focus our discussion of data gathering elements in this 

PQAPP on electronic questionnaires. In addition, for small, directed data collection efforts, such 

as to collect pricing or other information from vendors, EPA may choose to conduct telephone 

surveys. In these cases, Abt Associates will gather responses onto a data collection form 

developed specifically for the purpose. We describe our general procedures for conducting 

primary data gathering interviews in Section 4.3.

3

 

3

In the unlikely event that larger collection efforts need to be conducted by telephone, we will discuss the 

applicable QA procedures in a project-specific SQAPP. For example, Abt Associates would program the data 

collection form into Computer Assisted Telephone Interviewing system (CATI). CATI minimizes interviewing 

error, through its automatic skip patterns and range checks, which has the added benefit of minimizing post 

field editing. Specific data collection protocols for phone interviews will be determined based on a discussion 

with the EPA work assignment manager and will be addressed in the SQAPP.
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 61

 Procedures for handling CBI responses. We will handle any technical and/or financial 

information designated CBI in a questionnaire response or gathered during phone conversation 

according to Abt Associates’ CBI security plan. The potential for the primary data to be classified 

as CBI will affect the questionnaire format, methods for transmitting data to EPA and Abt 

Associates, and subsequent processing of the data in the analyses.

Abt Associates will work with the EPA WAM, the EPA statistics team, and any other relevant contractors 

(e.g., for engineering analysis) to develop the primary data collection process. Generally, the EPA 

statistics team develops the survey sample and weights, and provides these data to Abt Associates for use 

in implementing the survey. Abt Associates may assist EPA in defining the sampling frame – the list of 

all entities that can be sampled – based on the goal of the collection efforts by providing information on 

the number and characteristics of regulated entities. For example, the sampling frame may be stratified by 

industry sector, geographic area, or other characteristics (fuel type, economic activity performed, number 

of employees, etc.). Abt Associates will rely on publicly available data (i.e., based on existing data 

described in Section 3) to support EPA in identifying the appropriate sampling frame for industry 

surveys. In past primary data collection efforts, Abt Associates also helped EPA develop the data 

collection instrument (questionnaire), supported development of the Information Collection Request 

(ICR) as needed, provided helpline support to data providers, and reviewed and processed the primary 

data. For smaller data collection efforts involving fewer than ten respondents, Abt Associates identifies 

the entities based on the purpose of the request, collects the information, and summarizes the results for 

use in the analyses. 

We will document the data collection process in a series of planning memoranda and/or documents, 

including input for developing any required ICR Supporting Statement.

4.1.2 Data Gathering Methods and Steps

Data gathering methods and steps will depend on the type of data to be gathered (facility information, 

pricing data) and Abt Associates’ role in the effort.

4 Below, we describe general QA-related 

considerations and procedures for industry surveys and other primary data collection efforts carried out by 

Abt Associates to support ELG development. We will describe additional procedures that are specific to a 

given project or to other types of primary data collection efforts, such as stated-preference surveys – in a 

SQAPP.

 Identify information to be collected and purpose: During the design phase, Abt Associates 

develops a list of data to be collected from the respondents, based on the analytic blueprint and 

data gaps identified during review of available secondary data. We also outline alternative to 

using primary data, if any. We then identify the desired data resolution (facility/immediate 

owner/parent owner, time period, budget component, etc.), units, additional considerations that 

could affect the way the data will be collected (e.g., Are the respondents likely to have the 

requested information at the desired level of detail? Are categories or ranges sufficient or do we 

need actual values?). We describe the results of this assessment in a memorandum to the EPA 

WAM for review and approval prior to proceeding with developing the survey instrument. 

 Design instrument to minimize burden and potential for errors: Abt Associates designs 

questionnaires to collect the necessary information while minimizing respondent burden and 

 

4

For some industry surveys, Abt Associates may be conducting only some of the activities enumerated. For 

example, for the Steam Electric industry survey, a different contractor was responsible for mailing and 

receiving the surveys. Abt Associates supported preparation of the questionnaire and sample frame, and 

received and processed the economic and financial sections of the completed survey. 
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 62

potential for error. We review the literature and secondary data collection instruments to identify 

industry-specific terms that would be familiar to the respondents. We develop electronic forms 

that incorporate validation rules to minimize the potential for invalid entries and ensure data 

consistency. For example, we may use Excel forms to request information from industry 

respondents and use the built-in data validation feature to limit entries to a pre-populated list (e.g., 

NAICS codes, state abbreviations, pre-determined ranges).

 QA and test instrument: We pretest the questionnaire using staff not involved in the development 

of the questionnaire and, if feasible and approved by EPA, using respondents selected to 

approximate the target population (e.g., trade association or industry representatives). Pre-testing 

the survey instrument validates the adequacy, clarity, and balance of the data collection 

instruments, both for interviewers administering the questionnaire (if relevant) and the respondent 

answering the questions. 

 Obtain the respondent list. As noted above, Abt Associates will typically receive the survey 

sample from the EPA statistics team. If needed, Abt Associates’ sampling statisticians can 

provide support in developing a sample design that meets the project’s analytic objectives and 

resources. If needed, we will detail this activity in a project-specific SQAPP. 

 Develop survey tracking database: Once EPA has determined the final sample, Abt Associates 

develops a tracking database of respondents in Excel, Access or other similar software. We use 

this database throughout the survey process to track status and actions related to each response 

(e.g., survey sent, response received, response reviewed, follow-up, QA completed, etc.). 

 Mailing: For mail surveys targeting large samples, Abt Associates uses a three-step process to 

maximize the response rate (as specified in Dillman et al., 2009): (1) send a pre-alert letter 

announcing the forthcoming survey, explaining its purpose and process; (2) mail the survey 

package, which includes a cover letter, questionnaire, and business reply return envelope; and (3) 

follow up to increase the response rate. To maximize response rates, it is essential to allot at least 

8 weeks for survey administration; avoid scheduling mailings during a holiday season (especially 

between Thanksgiving and Christmas); and limit the length of the survey.

 Follow up with respondents to increase timeliness and response rate: Industry survey 

respondents are typically compelled to return completed questionnaires within the deadlines 

specified by EPA. We track responses received against the list of respondents that were sent a 

survey. As the deadline approaches, we contact respondents to ensure that they are able to 

complete the survey on time. For other surveys where participation may be voluntary, our typical 

approach includes making at least three additional re-contact attempts after an initial contact 

failure. 

 Setup and maintain telephone and email helpline to answer respondent questions: Throughout the 

survey implementation period, Abt Associates set up and maintain a dedicated toll-free telephone 

number and email account to answer questions on the survey. We enter each call and email into a 

tracking sheet and record the date/time, survey respondent ID, contact information, details of the 

question, name of Abt Associates employee answering, and resolution.

4.1.3 Completed Questionnaire Handling and Custody

Abt Associates may receive completed questionnaires directly from the respondents, from EPA, or from 

another contractor (e.g., engineering contractor). Upon receiving a completed questionnaire on CD-ROM

or other electronic media, we log the receipt in our tracking database, verify that the file is readable, and 

acknowledge receipt to the sender. 
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 63

We maintain material in their “as received” state (e.g., original media), and create electronic copies of the 

files in the project folder for further review and analysis (see Section 3.1.6). We store the material in 

accordance with any applicable CBI procedures as detailed in the CBI Plan for Contract EP-C-13-039. 

4.1.4 Data Management

We describe general data management procedures below; specific data management procedures that vary 

depending on the type of primary data collected may be covered in a work assignment specific SQAPP.

Abt Associates typically receives industry survey responses as separate files for each respondent (e.g., 

individual Excel workbook for each plant). Abt Associates consolidates these data into a Draft Database 

to facilitate review and use in eventual analyses. Depending on the number of responses and complexity, 

we manage the consolidated data using spreadsheet or relational database software (Excel, Access). For 

example, for the Steam Electric industry survey, Abt Associates compiled the information from over 700 

individual files into a Microsoft Access database. We used the database to organize the survey data into a 

series of indexed tables, and enforced relationship and integrity checks to ensure data consistency at the 

plant, owner, and unit level. Indexes and relational integrity checks ensures that the data are internally 

consistent (e.g., information is exactly the same across tables, where applicable), and help prevent 

duplicate records, for example by ensuring that only one record exists for each unique respondent or 

combination of respondent and other parameter (e.g., plant-year).

We include a data dictionary and metadata in the database to clearly define the content of each table and 

field, and document the source of the data (file name, dates created/imported/modified).

We transfer data from individual respondent files to the consolidated data set using automated data 

uploads. For all but the simplest data transfer,5this step entails transferring electronic data from the 

original source (e.g., Microsoft Excel spreadsheet) into the appropriate database tables using customized

import programs. Using customized programs ensures consistency and reproducibility, and minimizes the 

potential for transmission errors. We develop and test the data transfer program using a sample 

questionnaire to ensure that it provides error-free transcription of the raw data, as verified by checking 

each and every imported data against the original source and reviewing the error log (e.g., error 

automatically generated by Access when encountering a data type mismatch such as importing text into a 

numeric field). We adjust the import program if needed, and re-test and verify the transfer until 

100 percent of the test data are correctly uploaded into the final data set and no errors are generated. We 

review the log for each individual response file imported into the database to verify that no errors were 

generated. Two important considerations for transferring data include:

 Addressing potential data type mismatch: We account for cases in which respondents may 

provide data in different formats (e.g., number for actual values or text to indicate that the field is 

not applicable) by either mapping the data to a text-formatted field, which is the most flexible 

data type, or flagging the data point in the log for manual review.

 Differentiating between blanks and zeros: We translate any blank response into Null whereas we 

store zero values as 0. We further evaluate Null data during QA review (see below) to identify 

responses requiring follow-up to fill data gaps versus cases in which the respondent had a valid 

reason for leaving the field blank (i.e., circumstances did not apply).

 

5 A simple data transfer not involving the use of a customized program includes transfer from one file or table to 

a single database table having the same format and fields. Such simple data transfers are not common for 

industry surveys for which we need to combine answers from multiple respondents and several different types 

of data.
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 64

4.1.5 Assessment and Response Actions

We assess the processes and data on an ongoing basis throughout the data gathering phase to ensure that 

the primary data collection effort will meet the desired objectives. In the early phases, before the survey is 

mailed to respondents, response actions involve revising draft documents to address comments or 

reviews. During the response period for the survey, any issue identified with the questionnaire design 

(e.g., multiple respondents contacting the hotline with similar questions) prompts discussion with the 

EPA WAM on the best way to address the issue and ensure smooth and accurate data collection. This 

may involve developing a “Frequently Asked Questions” (FAQ) document with clarification on 

questions, releasing an alternate version of the survey, or other appropriate response actions agreed to 

with the EPA WAM. 

Once survey responses come in, we conduct a preliminary review to detect any systematic issue with the 

data that may require adjustments to the data compilation procedures. If we identify problems, we quickly 

report them to the EPA WAM, along with the implications for uses of the data and recommended 

response actions. 

Upon completing the transfer of individual survey responses to the database, we check that the number of 

records matches the number of survey responses processed, and again look for trends in the data 

completion rates to highlight any question that respondents may have systematically misunderstood or 

skipped. If we identify problems, we quickly report the issue to the EPA WAM, along with the

implications for uses of the data and recommended response actions.

4.1.6 Reports to Management

We track progress in uploading and reviewing responses using the tracking database, and provide regular 

updates to EPA on the number of responses that have been received and processed. As described above, 

once the individual responses have been compiled in to the database, we summarize the types of errors

and error rate for different data elements, including missing data. We also highlight any other issues with 

eventual use of the data, such respondents returning surveys indicating that they should not have had to 

respond, which may require recalculation of the sample weights.

4.2 Data Quality Assessment and Use

This section describes considerations for reviewing primary data prior to use in conducting analyses, but 

after we have obtained and compiled the primary data. 

4.2.1 Data Review, Verification, and Validation

This stage involves a more detailed review of the data (building on the preliminary review conducted 

during the earlier stage to verify that the data gathered in the database match the original sources). A first 

step involves going through Null entries to differentiate between valid cases (did not apply to the 

respondent) versus requires follow up with the respondent.

6 A second step involves going through the 

comment fields to catch any additional insight on the data. 

At this stage, we also standardize records, verify equalities, and detect outliers through additional steps 

that include:

 Verifying information across data elements. This first level of review seeks to verify that the 

information is consistent across data elements for a given respondent. This includes checking that 

 

6 We seek to minimize the frequency of this case through survey instrument design that allows for the respondent 

to enter N/A when the question is not applicable.
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 65

expected relationships and equalities are met. For example, the sum of generation by fuel type 

should equal total reported generation.

 Verifying information across sections. This second level of review seeks to verify that the 

information is consistent or not inconsistent across sections for a given respondent. For example, 

a plant that reported no electricity generation in one year in the section that addresses economic 

activity, should also report no revenue from sale of electricity in the section that addresses 

sources of revenue. 

 Verifying data across the survey population. This third level of review seeks to detect 

discrepancies by comparing responses across respondents. This entails comparing responses that 

should be the same (e.g., responses for two plants owned by the same corporate entity should 

report the same revenue for that entity). It also entails summarizing and comparing responses 

across respondents to identify apparent outliers (outside the interquartile range) in the reported 

data or in derived metrics such as revenue per unit of activity (e.g., identify cases in which a plant 

has unit revenue widely different from that for other plants). 

 Validating data against information reported in public records. This fourth level of review 

compares the survey responses to data available publicly, when possible. Since the purpose of 

industry survey is to collect data that are not available from existing public data, this comparison 

is generally possible for select data items only or for industry totals (e.g., comparing plant-level 

electricity generation values to data reported by EIA, comparing industry revenue to Economic 

Census values, comparing revenue of publicly traded companies to U.S. Securities and Exchange 

Commission (SEC) Form 10-K, which are generally available from the company websites). 

If our review highlights issues with an individual respondent’s information (e.g., values provided are an 

order of magnitude different from those reported for other plants, potentially suggesting that the units of 

measure are not those indicated), we contact the respondent to verify the data. We also follow up on 

incomplete data, comments that are unclear, and other issues we identified during the review. In some 

instances we obtain data from public records such as SEC filings. We do not overwrite original data 

during the QA review and validation, but instead record any revisions and final values in separate fields, 

along with the reason for the change.

4.2.2 Reconciliation with User Requirements

A final stage in the assessment is to review the survey data, after the review and validation steps have 

been completed, to assess if the data will meet the analytic needs or if additional steps will be required to 

provide the needed inputs. This assessment focuses on the data elements that are absolutely needed to 

support the analyses. 

When select survey data are not available for a given plant or owner and no additional information can be 

obtained from the respondent, Abt Associates will discuss with the EPA WAM appropriate methods for 

imputing needed values based on available data from the survey or from secondary data sources discussed 

in Section 3. For example, for Steam Electric, there were too many gaps and consistency issues in the 

plant revenue data reported in the survey to use the data directly. Instead, we used EIA data to impute 

plant-level revenue based on electricity sales (consistently across all plants), and used the survey data to 

validate the EIA-based estimates. Other data imputation methods include using average values of data 

reported for other plants or owners (e.g., using average fuel costs for other plants within the same electric 

power market region) or estimating missing values based on other information available for the plant or 

owner (e.g., allocating revenue to the plant in proportion to the quantity of electricity generated). 

We document our assessment of the survey database in a memorandum to the EPA WAM, noting any 

concerns with the completeness or representativeness of the data. This includes documenting incomplete 
Primary Data Collection

Abt Associates PQAPP EP-C-13-039 July 2014 66

and missing survey responses, and responses indicating that a plant should not have been included in the 

sample frame (e.g., no longer in business), so that the EPA statistics team can recalculate the survey 

weights as needed. 

Abt Associates may also need to calculate different survey weights to support costs and/or benefit 

analyses because sample weights that are based only on facility-specific (e.g., engineering) characteristics 

and various non-facility factors can lead to a conditional bias in the estimation of national-level benefits. 

For example, important non-facility factors that influence the occurrence and size of benefits include

characteristics of the receiving waters, number of regulated facilities discharging to the affected 

waterbody, and benefitting population. Abt Associates will develop a post-stratification weightadjustment method (e.g., raking) to account for additional variables that were not accounted for in the 

original sample design and that may affect benefit occurrence. We would describe these methods in 

technical memoranda to the EPA WAM, and in the technical support documents for the rulemaking (e.g., 

Benefits Analysis Report).

4.3 Telephone Interviews

Abt Associates uses telephone interviews to collect information from small number of respondents. Such

interviews are conducted, for example, to gather prices from equipment vendors or laboratories to support 

estimates of compliance costs, or to understand certain aspects of a study or process.

Steps involved in conducting the interviews include:

 Identifying the needs for the information and the target respondents. In this first step, we work 

with the EPA WAM to establish the primary data needs based on the analytic blueprint and 

review of the secondary data, we also identify the entities that may be able to provide information 

using web searches, professional contacts, etc., agree on a number of contacts (not to exceed 

nine) that will be made to collect the information, and process for contacting the respondents 

(e.g., email to screen and schedule interviews). The number of contacts will be based on the 

expected variability in the information provided; we generally consider three estimates to be 

appropriate for stock items. 

 Developing the list of questions/interview guide. When looking for very simple data such as the 

costs of standard items (e.g., laboratory analysis following a standard method), the interview 

guide consists of a simple script that introduces the interviewer, purpose for the call, and 

describes the request. For more complex data collection where questions may evolve depending

on initial answers, the interview guide includes potential follow up and scenarios. 

 Gathering data. Upon approval by the EPA WAM of the target respondents and interview 

process, the interviewer contacts each of the target respondents and collects the information 

following the script established in the interview guide. We record the answers along with the 

date, time of the interview and contact information.

 Summarizing the information. In this final step, we prepare a memorandum compiling the data 

across the interviews, presenting the distribution of values and recommendations for the use of 

the data in the analyses.
References

Abt Associates PQAPP EP-C-13-039 July 2014 67

5. References

Bateman, I.J. and A.P. Jones. 2003. Contrasting Conventional with Multi-Level Modeling Approaches to 

Meta-Analysis: Expectation Consistency in U.K. Woodland Recreation Values. Land Economics 79(2): 

235-258. 

Bergstrom, J.C. and P. De Civita. 1999. Status of Benefits Transfer in the United States and Canada: A 

Review. Canadian Journal of Agricultural Economics 47(1): 79-87. 

Bergstrom, J.C. and L.O. Taylor. 2006. “Using Meta-Analysis for Benefits Transfer: Theory and 

Practice.” Ecological Economics 60(2):351-360. 

Boyle, K. J., Parmeter, C. F., Boehlert, B. B., & Paterson, R. W. 2013. Due diligence in meta-analyses to 

support benefit transfers. Environmental and Resource Economics, 55(3), 357-386.

California Environmental Protection Agency, Office of Environmental Health Hazard Assessment (Cal 

OEHHA). 2013. Legget Model Code. Available at: http://oehha.ca.gov/air/legget.html.

Dalhuisen, J.M., R.J.G.M. Florax, H.L.F. de Groot, and P. Nijkamp. 2003. Price and income elasticities of 

residential water demand: A meta analysis. Land Economics 79(2):292-308. 

Dillman, D.A., Smyth, J.D. and Christian, L.M. 2009. Internet, Mail, and Mixed Mode Surveys: The 

Tailored Design Method (3rd ed.). New York, NY: John Wiley and Sons.

Glass, G.V. 1976. Primary, Secondary, and Meta-Analysis of Research. Educational Researcher 5(10): 3-

8. 

Johnston, R.J., M.H. Ranson, E.Y. Besedin and E.C. Helm. 2006. “What Determines Willingness To Pay 

Per Fish? A Meta-Analysis of Recreational Fishing Values.” Marine Resource Economics 21:1-32. 

Johnston, R.J., E.Y. Besedin, R. Iovanna, C.J. Miller, R.F. Wardwell and M.H. Ranson. 2005. 

“Systematic Variation in Willingness to Pay for Aquatic Resource Improvements and Implications for 

Benefit Transfer: A Meta-Analysis.” Canadian Journal of Agricultural Economics 53(2-3):221-248. 

Johnston, R.J., J.J. Opaluch, T.A. Grigalunas, and M.J. Mazzotta. 2001. Estimating Amenity Benefits of 

Coastal Farmland. Growth and Change 32(summer): 305-325. 

Leggett R.W. 1993. An age-specific kinetic model of lead metabolism in humans. Environ Health 

Perspect 101(7): 598-616.

Moeltner, K., K.J. Boyle, and R.W. Paterson. 2007. “Meta-Analysis and Benefit Transfer for Resource 

Valuation—Addressing Classical Challenges with Bayesian Modeling.” Journal of Environmental 

Economics and Management 53(2):250-269. 

Navrud, S. and R. Ready. 2007. Review of Methods for Value Transfer. In S. Navrud and R. Ready 

(eds.), Environmental Value Transfer: Issues and Methods. Dordrecht, The Netherlands: Springer. pp.1-

10. 

Poe, G.L., K.J. Boyle, and J.C. Bergstrom. 2001. A Preliminary Meta Analysis of Contingent Values for 

Ground Water Quality Revisited, in J.C. Bergstrom, K.J. Boyle and G.L. Poe, eds. The Economic Value 

of Water Quality. Northampton, MA: Edward Elgar Publishers. 

Rosenberger, R.S. and R.J. Johnston. 2007. Selection Effects in Meta-Valuation Function Transfers. 

Meta-Analysis in Economics Conference, September 27-30, 2007, Sandbjerg Manor, Sønderborg, 

Denmark. 

Rosenberger, R.S. and J.B. Loomis. 2000. Using Meta-Analysis for Benefit Transfer: In-Sample 

Convergent Validity Tests of an Outdoor Recreation Database. Water Resources Research 36(4): 1097-

1107. 
References

Abt Associates PQAPP EP-C-13-039 July 2014 68

Santos, J.M.L. 1998. The Economic Valuation of Landscape Change. Cheltenham, UK: Edward Elgar. 

Smith, V.K., and L. Osborne. 1996. Do Contingent Valuation Estimates Pass the Scope Test? A Meta 

Analysis. Journal of Environmental Economics and Management 31(3): 287-301. 

Stanley, T. D., Doucouliagos, H., Giles, M., Heckemeyer, J. H., Johnston, R. J., Laroche, P., & Rost, K. 

2013. Meta-analysis of economics research reporting guidelines. Journal of Economic Surveys, 27(2), 

390-394.

United States Energy Information Administration (EIA). 2009. Form EIA-860 Detailed Data.

United States Environmental Protection Agency (EPA). December 2010. Guidelines for Preparing 

Economic Analyses.

United States Environmental Protection Agency (EPA). July 2010. EPA’s Action Development Process: 

Interim Guidance on Considering Environmental Justice During the Development of an Action.

United States Environmental Protection Agency (EPA). October 2009. Technical Support Document for 

the Preliminary 2010 Effluent Guidelines Program Plan. EPA 821-R-09-006.

United States Environmental Protection Agency (EPA). November 2006. EPA’s Action Development 

Process. Final Guidance for EPA Rulewriters: Regulatory Flexibility Act.

United States Environmental Protection Agency (EPA). 2006. Peer Review Handbook, 3rd Edition. 

Prepared for EPA’s Science Policy Council by Member of the Peer Review Advisory Group. EPA/100/B06/002.

United States Environmental Protection Agency (EPA). 2002. Guidelines for Ensuring and Maximizing 

the Quality, Objectivity, Utility, and Integrity, of Information Disseminated by the Environmental

Protection Agency. EPA/260R-02-008.
Appendix A 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌69

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Appendix A: Checklists

Table A-1: Document QA/QC Checklist

General Information

Work Assignment Number: 

Deliverable: 

Review Level/Reason (including quick turnaround): 

Work Assignment Manager : 

Project Staff:

Project Quality Assurance Advisor:

Due date:

Document QA/QC Checklist

Place an "x" in the appropriate check box followed by the name of the reviewer and date reviewed.

Level 1 Yes No N/A Reviewer Date Comments

Outline reviewed by WAM (and PQA Advisor if 

appropriate).

Detailed review by author and peer (or WAM).

Cited numbers match underlying sources.

Principal sources cited; references included.

Notes principal limitations.

Describes QA activities, per QAPP.

Document is free of grammatical errors.

Document uses established formatting style.

Level 1 Completed:
Appendix A 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌70

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Document QA/QC Checklist

Place an "x" in the appropriate check box followed by the name of the reviewer and date reviewed.

Level 2 (includes Level 1) Yes No N/A Reviewer Date Comments

Detailed review by WAM for accuracy, clarity of 

presentation, and completeness.

Detailed review by PQA Advisor or his/her designee, 

including for compliance with QAPP.

All sources cited; references included.

Discusses limitations, biases, and uncertainties in 

separate section.

Optional review by editor.

Optional formatting review.

Level 2 Completed:

Level 3 (includes Levels 1 and 2) Yes No N/A Reviewer Date Notes

Editorial and formatting review (clarity of 

presentation, consistency of formatting, etc.).

Final acceptance by Project Manager or his/her 

designee, including verifying use of appropriate 

language and characterization of the results.

Level 3 Completed:
Appendix A 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌71

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Table A-2: Spreadsheet QA/QC Checklist

General Information

Work Assignment Number:

Deliverable:

Review Level/Reason (including quick turnaround):

Work Assignment Manager :

Project Staff:

Project Quality Assurance Advisor:

Due date:

Spreadsheet QA/QC Checklist

Add additional rows for additional sheets when necessary. Place an "x" in the appropriate check box followed by the reviewer and date reviewed.

File General Yes No N/A Reviewer Date Comments

Formulas/calculations correctly reflect general 

methodology and assumptions

Results have been replicated/ reproduced by 

external calculation. Provide information on software 

and methodology used.

Results follow expected direction and approximate 

magnitude of change across analyzed cases (e.g., 

regulatory options)

References are appropriately cited

Assumptions, inputs, data, and outputs are shown 

on separate sheets. 

Hard-coded inputs or inputs calculated from hardcoded values only in Assumptions or Data sheets.

Sheet 1 Yes No N/A Reviewer Date Comments

Data are read in correctly/data sources are verified 

and documented

Formulas/calculations correctly reflect general 

methodology and assumptions

Formulas are correctly written

Formulas refer to the appropriate cells

Multiple steps in a calculation are shown in their own 

cell/row/column, to the extent practicable.
Appendix A 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌72

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Spreadsheet QA/QC Checklist

Add additional rows for additional sheets when necessary. Place an "x" in the appropriate check box followed by the reviewer and date reviewed.

Sheet 2 Yes No N/A Reviewer Date Notes

Data are read in correctly/data sources are verified 

and documented

Formulas/calculations correctly reflect general 

methodology and assumptions

Formulas are correctly written

Formulas refer to the appropriate cells

Multiple steps in a calculation are shown in their own 

cell/row/column, to the extent practicable.

Table A-3: Programming/Modeling QA/QC Checklist

General Information

Work Assignment Number:

Deliverable:

Review Level/Reason (including quick turnaround):

Work Assignment Manager :

Project Staff: 

Project Quality Assurance Advisor:

Due date:

Programming/Modeling QA/QC Checklist

Place an "x" in the appropriate check box followed by the reviewer and date reviewed.

Yes No N/A Reviewer Date Comments

Program has proper documentation or a "read me" 

file which adequately explains data sources, 

program functions, and outputs.

Data are read in correctly.

Program code is written correctly and contains no 

errors.

Overall program process is consistent with the 

results produced

Results can be independently replicated/reproduced 

with another program.
Appendix B

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌73 

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Appendix B: QA Audit Checklist

Project Name:

Work Assignment:

Work Assignment Manager:

Auditor:

Date:

Instructions:

1. Obtain the project work plan, QAPP, and any other planning documents from the Project Manager.

2. Review the QA requirements.

3. Interview the Project Manager and other team members, if necessary.

B.1 Planning and Project Management

Quantitative Review (to be completed by auditor after review of project files or discussion with 

appropriate team members)

1. What planning documents are available for this project?

☐QAPP

☐Work Plan

☐Project Instructions

☐Required by client

☐Required by client

☐Required by client

☐Provided to client

☐Provided to client

☐Provided to client

☐Reviewed by team

☐Reviewed by team

☐Reviewed by team

2. Were there any changes to the QAPP during the project, and if so, did the WAM distribute the changes

to the project team and client? 

☐Yes ☐No ☐N/A

3. Did the client request any QA reports? If so, did we provide them? 

☐Yes ☐No ☐N/A

4. Did the project team, PQA, or client identify any quality deficiencies during the project? If so, specify.

☐Yes ☐No ☐N/A

B.1 QA Specifications

1. Are there QA specifications for the project in the QAPP, Work Plan, or other key document? (Check 

all that apply)

☐Conceptual review (Complete Question 2) ☐Calculation review (Complete Question 3)

☐Deliverable review (Complete Question 3) ☐Senior review (Complete Question 6)

☐Peer review (Complete Question 6) ☐Other review (specify):
Appendix B 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌74

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Conceptual Review

2. Was there a conceptual review for planning major activities/deliverables? ☐Yes ☐No

If yes, does the reviewer have enough experience and a relevant degree? ☐Yes ☐No

Typically conceptual reviewers are at least at Professional Level 3, and often are the project manager or 

senior reviewer. Ask the project manager if there are other reviewer qualifications specified (e.g., 

someone outside the project, professional engineer, Professional Level 4, etc.).

Developmental Review

3. How are calculation and deliverable reviews documented?

Sign-off Sheet ☐Yes, always ☐Yes, mostly ☐Sometimes ☐No

Tracking Database ☐Yes, always ☐Yes, mostly ☐Sometimes ☐No

Other, explain:

4. Do subcontractors prepare deliverables?

☐Yes ☐No ☐N/A

5. If yes to question 4, what is the review process for their work products?

Senior and Peer Review

6. Was there a senior or peer review of major activities/deliverables?

☐Yes ☐No ☐N/A

7. If yes to question 6, who is the reviewer? Does the reviewer have enough experience and a relevant 

degree? Typically senior and peer reviewers are at least Professional Level 3. Ask the project manager if 

there are other reviewer qualifications specified (e.g., someone outside the project, professional engineer, 

Professional Level 4, etc.).

Obstacles to Review Process

8. According to the PM or other team members, what were the obstacles to performing any level of 

review?
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌75

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Appendix C: Meta-Analysis Data and Methods

One of the benefits transfer approaches that Abt Associates employs in supporting EAD in developing 

ELGs is meta-analysis. Abt Associates is developing two meta-analyses: (1) surface water quality 

valuation studies and (2) hedonic property studies of open space values. This appendix describes the data 

gathering (including specific sources), assessment, and use pertaining to estimating the meta-regression 

models, and reporting the resulting values.

The meta-analysis of surface water quality valuation studies is a re-examination and update of an existing 

meta-analysis that Abt Associates developed for EPA. While the existing meta-analysis model has been 

peer reviewed, and has been useful for cost-benefit analyses for multiple rulemakings, the reexamination 

is necessary to incorporate recent advances in meta-analysis, address limitations of the existing models 

(i.e., the inability to capture the effects of geographic scale of affected resources and the availability of 

regional substitutes) and improve the model for future rulemaking support. The reexamination includes a 

literature review to identify additional studies to include in the meta-data, and development and 

compilation of additional variables to include in a revised WTP function with a focus on geographic 

characteristics of affected resources and benefitting populations (e.g., geographic area covered by the 

study, size of the affected aquatic resource). Revisions to this PQAPP may be necessary once reexamination is complete, and there has been peer review of the revised meta-analysis.

C.1 Identifying Candidate Studies

Following the literature search procedures described in the body of this PQAPP (see Section 3.1.2), we 

will search for stated or revealed preference studies (e.g., hedonic implicit price models) using the 

following specific terms to identify valuation studies of surface water quality, and hedonic studies of open 

space values, to be sufficiently broad so not to miss relevant studies:

 Water quality, water clarity, nutrient removal/improvement, water quality index/ladder, clean 

water, water pollution reduction, habitat improvement, stream flow

 Open space, low impact development (LID), BMP, green infrastructure, conservation 

subdivision, environmental site design (ESD), property value, property price, WTP, hedonic, nonmarket values, residential subdivision, homeowner preferences.

In compiling meta-data, we are following Stanley et al. (2013) guidelines for documenting and reporting 

meta-regression analyses in economics, including documentation of the new literature search. This 

documentation includes: (a) the exact databases and other sources searched, (b) the precise combination 

of keywords, and (c) date completed. 

To be included in the metadata, candidate studies must be both commodity consistent and welfare 

consistent (Bergstrom and Taylor 2006). For commodity consistency, the amenity valued should be 

similar across studies in the metadata. Studies must also provide sufficient detail on the amenity itself as 

discussed by Johnston et al. (2005): “the study must provide sufficient information regarding resource, 

context, and study attributes to warrant inclusion in the meta-data.” Following the requirement for welfare 

consistency, the metadata will not combine studies representing theoretically distinct welfare measures

(i.e., stated and revealed preference data).

Candidate studies for the water quality and hedonic meta-analyses must also meet criteria for study

location and quality. All studies must be conducted in the United States or Canada, and must apply 

research methods that are supported by the literature and that provide WTP estimates consistent with 

neoclassical welfare theory.
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌76

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

We will also take steps to identify duplicative studies. Some studies may be released in multiple forms, 

such as working papers, conference papers, journal articles, and book chapters. We will screen articles to 

remove duplicative analysis and analyses that were subsequently revised and published in peer reviewed 

journals. Multiple values can be included for a study if they address a different component of the study 

data (e.g., a different watershed or affected population) or estimate values for alternative resource 

changes.

Sections C.4 and C.5 list the data sources we identified from these searches that meet the screening 

criteria described in the body of this PQAPP (see Section 3.1.3).

C.2 Estimating the Models

Glass (1976) characterizes meta-analysis as “the statistical analysis of a large collection of results for 

individual studies for the purposes of integrating the findings.” Meta-analysis is largely an empirical, 

data-driven process, but one in which variable and model selection is guided by theory (Bergstrom and 

Taylor 2006). Given a reliance on information available from the underlying studies that comprise the 

metadata, meta-analysis models most often represent a middle ground between model specifications that 

would be most theoretically appropriate and those specifications that are possible given available data. 

Abt Associates follows the mechanics of specifying and estimating meta-equations in resource economics 

applications as outlined in Smith and Osborne (1996), Rosenberger and Loomis (2000), Poe et al. (2001), 

Bateman and Jones (2003), Dalhuisen et al. (2003), Johnston et al. (2005, 2006), Bergstrom and Taylor 

(2006), Moeltner et al. (2007). More specifically, Abt Associates follows work by Bateman and Jones 

(2003), and Johnston et al. (2005), in applying a multilevel model specification to metadata, to address 

potential correlation among observations gathered from single studies. Also following prior work (e.g., 

Smith and Osborne 1996; Poe et al. 2001), we apply the Huber-White robust variance estimation. As 

described by Smith and Osborne (1996, p. 293), “this approach treats each study as the equivalent of a 

sample cluster with the potential for heteroskedasticity...across clusters.” 

To guide development of the model and variable specifications, we rely upon a set of general principles 

designed to help prevent excessive data manipulations and other factors that may lead to misleading 

model results. All else being equal, the general principles include:

 Fewer and simpler data transformations are better than more extensive transformations

 In the absence of overriding theoretical considerations, continuous variables are generally better 

than discrete variables derived from underlying continuous distributions

 Models should capture elements of the scope and scale of resource changes (e.g., the size of the 

affected waterbodies) as well as availability of substitute sites 

 Models should distinguish WTP associated with different types of resources and resource uses, 

particularly when relevant to the policy question at hand

 Following the “weak structural utility theoretic” (WSUT) approach of Bergstrom and Taylor 

(2006, p. 352), we avoid exogenous structural constraints to afford the flexibility necessary to 

appropriately model empirical patterns that may not necessarily flow from an underlying 

theoretical modeling structure.

In the case of the meta-analysis of surface water valuation studies, the revised model is being specified in 

order to capture four primary types of scope and scale effects on WTP, including the extent of the market 

analyzed by each primary study. The potential relevance of these effects is supported by theory and 

empirical findings in the primary valuation literature:

1. Scope of quality change (the magnitude of water quality change)
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌77

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

2. The geographical scale over which change occurs (the size of the water bodies affected by the 

change)

3. Extent of the market (the size of the sampled area, or geographic area over which WTP is 

quantified)

4. The proportion of affected versus unaffected resources (the size of affected water bodies relative 

to all water bodies in the sampled market)

We will develop scope and scale variables that are meaningful, measurable, and unambiguous. 

For each meta-analysis, we may estimate different model specifications and select the final model 

specifications using the following criteria: (1) fit, (2) intuitive results provided (including sign, statistical 

significance and correlations), and (3) common use in the empirical valuation and meta-analysis literature 

(e.g., Smith and Osborne 1996; Santos 1998; Johnston et al. 2001, 2005, 2006). Models are also evaluated 

based on correspondence of specifications to theory and prior expectations.

To test robustness of the parameter estimates from the model, we will conduct a set of procedures 

suggested by Boyle, et al. (2013), including tests for sample selection bias, horizontal robustness 

(inclusion/exclusion of observations or studies), and vertical robustness (inclusion/exclusion of variables).

It is possible that the literature does not include studies for some areas of the country and, as a result, the 

metadata could have better coverage for some areas of the country than others. Evaluation of geographic 

representativeness will be included mapping the location of studies included in the metadata. To assess 

potential sample selection bias we will rely on “due diligence” tests discussed by Boyle et al. (2013).

C.3 Documenting, Interpreting, and Presenting Results

We fully document the variables that we include and exclude in the regression models, along with the 

rationale. For example, we may exclude variable that are not statistically significant in any version of the 

model and for which there is no overriding rationale for retaining the variable in the model. For example, 

a variable distinguishing different types of discrete choice instruments (e.g., conjoint versus dichotomous 

choice) may add no significant explanatory power to the model. In comparison, we may retain certain 

variables for theoretical reasons, despite insignificance. Such specification of meta-analysis models using 

a combination of theoretical guidance and empirical considerations is standard in modeling efforts 

(Bergstrom and Taylor, 2006).

We also fully document our results, including variable; parameter estimate; and standard error. We review 

measures of equation fit, that the signs of significant parameter estimates correspond with intuition, and 

study level covariance factors. We may also compare our results to prior meta-analyses in the valuation 

literature. 

One of the primary means to assess the validity of benefit transfers—and the only one that may be applied 

in cases wherein the true value for the study site is unknown—is a value surface test (Bergstrom and De 

Civita, 1999). These tests involve assessments of ways in which “different factors may cause values to 

vary across sites, providing guidance for adjustments needed to make a valid transfer of value estimates 

from the study site(s) to the policy site” (Bergstrom and De Civita, 1999). We follow general approaches 

for such value surface assessments, which generally involve comparisons of empirical patterns found via 

meta-analysis to theoretical expectations or norms.

Results of meta-analysis are dependent on the sample of studies available for the given resource change 

(Navrud and Ready, 2007), and may be subject to various selection biases if the available literature does 

not provide a representative, unbiased perspective on welfare estimates associated with resource changes 

(Rosenberger and Johnston, 2007). However, we also take steps to ameliorate such potential biases, 

including the incorporation of both peer-reviewed and gray literature to avoid possible publication biases 

(Rosenberger and Johnston, 2007), and the use of a comprehensive literature review in the attempt to 
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌78

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

avoid—as much as possible—other types of selection biases. We also note when results should be viewed 

with caution.

The resulting preferred meta-analysis models will be peer reviewed according to EPA’s Peer Review 

Handbook (EPA 2006). Peer review is an important component of EAD’s quality control program,

providing a “process for enhancing a scientific or technical work product so that the decision or position 

taken by the Agency, based on that product has a sound, credible basis” (EPA 2006). The burden for 

implementing the peer review rests on EAD, not Abt Associates.

C.4 : Studies Used in Surface Water Valuation Meta-Analysis

Abt Associates is using the following studies in meta-analysis of surface water values, and will update 

this PQAPP with additional results of ongoing efforts as needed. 

Adamowicz, W.L., Louviere, J., and Williams, M. 1994, Combining Revealed and States Preference 

Methods for Valuing Environmental Amenities. Journal of Environmental Economics and Management. 

Vol (26), pp. 271-292, 1994.

Aiken, Richard A. 1985. Public Benefits of Environmental Protection in Colorado. Masters thesis, 

Colorado State University.

Anderson, G.D. and S.F. Edwards. 1986. Protecting Rhode Island’s Coastal Salt Ponds: An Economic 

Assessment of Downzoning. Coastal Zone Management Journal. 14( 1⁄2): 67-91.

Azevedo, Christopher, J.A. Herriges, and C.L. Kling. 2001. Valuing Preservation and Improvements of 

Water Quality in Clear Lake. Center for Agricultural and Rural Development (CARD), Iowa State 

University, Staff Report 01-SR 94.

Banzhaf, Spencer H, Dallas Burtraw, David Evans, and Alan Krupnick. 2006. "Valuation of Natural 

Resource Improvements in the Adirondacks." Land Economics, 82(3): 445-464.

Bockstael, N.E., K.E. McConnell and I.E. Strand. 1989. Measuring the Benefits of Improvements in 

Water Quality: The Chesapeake Bay. Marine Resource Economics. Vol. 6: 1-18.

Bockstael, N.E., McConnell, K.E., and Strand, I.E. 1988. Benefits from improvements in Chesapeake Bay 

Water Quality. Department of Agricultural & Resource Economics, University of Maryland, 1988.

Borisova, Tatiana, Alan Collins, Gerard D’Souza, Matthew Benson, Mary Leigh Wolfe, and Brian 

Benham, 2008. A Benefit-Cost Analysis of Total Maximum Daily Load Implementation. Journal of the 

American Water Resources Association (JAWRA) 44(4):1009-1023. DOI: 10.1111/j.1752-

1688.2008.00216.x

Breffle, William S., Morey, E., Rowe, R., Waldman, D. and Wytinck, S. 1999. Recreational Fishing 

Damages from Fish Consumption Advisories in the Waters of Green Bay. Prepared by Stratus 

Consulting Inc. for US Fish and Wildlife Service, US Department of Interior and US Department of 

Justice, 1999.

Brox, J., R. Kumar and K.R. Stollery. 2003. Estimating Willingness to Pay for Improved Water Quality 

in the Presence of Item Nonresponse Bias. American Journal of Agricultural Economics, Vol (85), pp. 

414-428, 2003.

Cameron, T.A. and D.D. Huppert. 1989. OLS versus ML Estimation of Non-market Resource Values 

with Payment Card Interval Data. Journal of Environmental Economics and Management. 17, 230-246.

Cameron, Trudy Ann. 1988. Using the Basic "Auto-validation" Model to Assess the Effect of 

Environmental Quality on Texas Recreational Fishing Demand: Welfare Estimates. EPA National Center 

for Environmental Economics, 1988.
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌79

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Carson, R., Mitchell, R., Hanemann, W. M., Kopp, R., Presser, S., Ruud, P. 1992. A Contingent 

Valuation Study of Lost Passive Use Values Resulting from the Exxon Valdez Oil Spill. A Report to the 

Attorney General of the State of Alaska, 1992.

Carson, Richard T., & Mitchell, Robert C, "The Value of Clean Water: The Public's Willingness to Pay 

for Boatable, Fishable, and Swimmable Quality Water," Water ResourcesResearch, vol. 29, 1993.

Carson, Richard T., W. M. Hanemann, R. J. Kopp, J. A. Krosnick, R. C. Mitchell, S. Presser, P. A. Ruud, 

and V. K. Smith. 1994. Prospective Interim Lost Use Value due to DDT and PCB Contamination in the 

Southern California Bight. Volume 2. 

Clonts, Howard A., and J.W. Malone. 1990. Preservation attitudes and consumer surplus in free flowing 

rivers. In: Social Science and Natural Resource Recreation Management, Joanne Vining, editor. 

Westview Press, Boulder, CO. Vol. pp. 301-317.

Collins, A. R., R. S. Rosenberger, et al. (2009). Valuing the restoration of acidic streams in the 

Appalachian Region: A stated choice method. Environmental economics for watershed restoration. H. W. 

Thurston, M. T. Heberling and A. Schrecongost. Boca Raton, CRC Press.

Collins, Alan R. and Rosenberger, Randall S., (2007), Protest Adjustments in the Valuation of Watershed 

Restoration Using Payment Card Data, Agricultural and Resource Economics Review, 36, issue 2.

Corrigan, J R, K J Egan, and J A Downing. 2009. "Aesthetic Values of Lakes and Rivers." Elsevier Inc.

Croke, Kevin, R.G. Fabian, and G. Brenniman. 1986-87. Estimating the Value of Improved Water 

Quality in an Urban River System. Journal of Environmental Systems. Vol. 16, No. 1. pp. 13-24.

Cronin, Francis J. 1982. Valuing nonmarket goods through contingent markets. Pacific Northwest 

Laboratory, PNL 4255, Richland, WA.

De Zoysa, A. D. N. 1995. A Benefit Evaluation of Programs to Enhance Groundwater Quality, Surface 

Water Quality and Wetland Habitat in Northwest Ohio. Dissertation, Ohio State University.

Desvousges, William H., Smith, V. Kerry and Fisher, Ann. 1987. Option Price Estimates for Water 

Quality Improvements: A Contingent Valuation Study for the Monongahela River. Journal of 

Environmental Economics and Management. Vol (14), pp. 248-267, 1987.

Desvousges, William H., V.K. Smith, and M.P. McGivney. 1983. A Comparison of Alternative 

Approaches for Estimating Recreation and Related Benefits of Water Quality Improvements. U.S. 

Environmental Protection Agency, Economic Analysis Division, Washington.

Douglas, A. J. and Sleeper, A. 2005. Estimating Recreation Trip Related Benefits for the Klamath River 

Basin with TCM and Contingent Use Data. 

Egan, K., Herriges, J., Kling, C. and Downing, J. 2004. Recreation Demand Using Physical Measures of 

Water Quality. Working Paper, Center for Agricultural and Rural Development, Iowa State University, 

2004.

Farber, S., and Griner, B. (2000). “Using Conjoint Analysis to Value Ecosystem. Change,” 

Environmental Science and Technology, 34(8), 1407-1412.

Farber, Stephen, and B. Griner. 2000. Valuing watershed quality improvements using conjoint analysis. 

Ecological Economics. 34 (1): 63 - 76.

Feather, P.M., Hellerstein, D. and Tomasi, T. 1995. A Discrete-Count Model of Recreation Demand. 

Journal of Environmental Economics and Management, Vol (29), pp. 214-227, 1995.

Greene, G., Moss, C. and Spreen, T. 1997. Demand for Recreational Fishing in Tampa Bay, Florida: A 

Random Utility Approach. Marine Resource Economics, Vol (12), pp. 293-305, 1997.
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌80

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Greenley, D.A., R.G. Walsh, and R.A. Young. 1981. Option Value: Empirical Evidence from a Case 

Study of Recreation and Water Quality. The Quarterly Journal of Economics, Vol (96), pp. 657-674, 

1981.

Hayes, K.M., T.J.Tyrell and G. Anderson. 1992. Estimating the Benefits of Water Quality 

Improvements in the Upper Narragansett Bay. Marine Resource Economics. Vol. 7, pp. 75-85.

Herriges, Joseph A., and J.F. Shogren. 1996. Starting Point Bias in Dichotomous Choice Valuation with 

Follow-Up Questioning. Journal of Environmental Economics and Management. Vol. 30, No. 1, pp. 

112-131.

Hite, D. 2002. Willingness to Pay for Water Quality Improvements: The Case of Precision Application 

Technology. Deaprtment of Agricultural Economics and Rural Sociology, Auburn University, Auburn, 

AL; 8 August 2002.

Hoehn, J., Tomasi, T., Lupi, F., and Chen, H. 1996. An Economic Model for Valuing Recreational 

Angling Resources in Michigan. Volume I. Report submitted to Michigan Department of 

Environmental Quality and Michigan Department of Natural Resources, 1996.

Holmes, Thomas P. & Bergstrom, John C. & Huszar, Eric & Kask, Susan B. & Orr, Fritz III, 2004. 

"Contingent valuation, net marginal benefits, and the scale of riparian ecosystem restoration," Ecological 

Economics, Elsevier, vol. 49(1), pages 19-30, May.

Huang, Ju-Chin, T. Haab, J.C. Whitehead. 1997. Willingness to Pay for Quality Improvements: Should 

Revealed and Stated Preference Data Be Combined? Journal of Environmental Economics and 

Management. Vol. 34, No. 3, pp. 240-255.

Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of 

Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU-TB042 2000, 1999.

Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of 

Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU-TB042 2000, 2000.

Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of 

Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU-TB042 2000, 2001.

Hushak, L. and M. Bielen. 1999. Valuing the Ottawa River: The Economic Values and Impacts of 

Recreational Boating. Ohio Sea Grant College Program Technical Bulletin Series Publication OSHU-TB042 2000, 2002.

Kaoru, Yoshiaki. 1993. Differentiating Use and Nonuse Values for Coastal Pond Water Quality 

Improvements. Environmental and Resource Economics. Vol. 3. pp. 487 494.

Lant, C. L. and R.S. Roberts. 1990. Greenbelts in the Cornbelt: Riparian Wetlands, Intrinsic Values, and 

Market Failure. Environment and Planning A, Vol. 22, pages 1375-1388.

Lant, C.L., and G.A. Tobin. 1989. The Economic Value of Riparian Corridors in Cornbelt Floodplains: A 

Research Framework. Professional Geographer Vol (41), pp. 337-349, 1989.

Lichtkoppler, F.R. and T.W. Blaine. 1999. Environmental Awareness and Attitudes of Ashtabula County 

Voters Concerning the Ashtabula River Area of Concern: 1996-1997. J. Great Lakes Resources, Vol 

(25), pp. 500-514, 1999.

Lindsey, G. 1994. Market Models, Protest Bids, and Outliers in Contingent Valuation. Journal of Water 

Resources Planning and Management, Vol (12), pp. 121-129, 1994.
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌81

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Lipton, D. 2004. The Value of Improved Water Quality to Chesapeake Bay Boaters. Marine Resource 

Economics. Vol(19):265-270.

Londoño Cadavid, C. and A. W. Ando (2013). "Valuing preferences over stormwater management 

outcomes including improved hydrologic function." Water Resources Research 49: 4114-4125.

Loomis, J., Paula Kent, Liz Strange, Kurt Fausch and Alan Covich. Measuring the total economic value 

of restoring ecosystem services in an impaired river basin: results from a contingent valuation survey. 

Ecological Economics, Vol (33), pp. 103-117, 2000.

Loomis, J., Paula Kent, Liz Strange, Kurt Fausch and Alan Covich. Measuring the total economic value 

of restoring ecosystem services in an impaired river basin: results from a contingent valuation survey. 

Ecological Economics, Vol (33), pp. 103-117, 2000

Loomis, John B. 1996. How Large is the Extent of the Market for Public Goods: Evidence from a Nation 

Wide Contingent Valuation Survey. Applied Economics. 28(7): 779-782.

Lyke, A.J. 1993. Discrete Choice Models to Value Changes in Environmental Quality: A Great Lakes 

Case Study. Dissertation submitted to the Graduate School of the University of Wisconsin, Madison.

Magat, Wesley A., J. Huber, K.W. Viscusi, and J. Bell. 2000. An Iterative Choice Approach to Valuing 

Clean Lakes, Rivers, and Streams. Journal of Risk and Uncertainty. Vol. 21, No. 1. pp. 7-43.

Matthews, Leah Greden, F.R. Homans, and K.W. Easter. 1999. Reducing Phosphorous Pollution in the 

Minnesota River: How Much is it Worth? Staff Paper. Department of Applied Economics, University of 

Minnesota.

Mazzotta, Marisa J. 1996. Measuring Public Values and Priorities for Natural Resources: An 

Application to the Peconic Estuary System. Ph.D. Dissertation, University of Rhode Island.

Mitchell, Robert Cameron, and R.T. Carson. 1981. An experiment in determining willingness to pay for 

national water quality improvements. Preliminary Draft of a report to the U.S. Environmental Protection 

Agency. Resources for the Future, Inc., Washin.

Morey, E., Breffle, W., Rowe, R. and Waldman, D. 2002. Estimating Recreational Trout Fishing 

Damages in Montana's Clark Fork River Basin: Summary of a Natural Resource Damage Assessment. 

Journal of Environmental Economics and Management, 66(2), pp. 159-170, 2002.

Needelman, M. and Kealy, M. 1995. Recreational Swimming Benefits of New Hampshire Lake Water 

Quality Policies: An Application of a Repeated Discrete Choice Model. Agricultural and Resource 

Economics Review, Vol (24), pp. 78-87, 1995.

Olsen, D., J. Richards, and R.D. Scott. 1991. "Existence and Sport Values for Doubling the Size of 

Columbia River Basin Salmon and Steelhead Runs. Rivers 2(1):44-56.

Patrick, R., J. Fletcher, S. Lovejoy, W. Van Beek, G. Hollowy, and J. Binkley. 1991. Estimating 

Regional Benefits of Reducing Targeted Pollutants: An Application to Agricultural Effects on Effects on 

Water Quality and the Value of Recreational Fishing. Journal of Environmental Management, Vol (33), 

pp. 301-310, 1991.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling 

the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781–1788, 1995.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling 

the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781–1788, 1996.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling 

the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781–1788, 1997.
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌82

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling 

the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781–1788, 1998.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling 

the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781–1788, 1999.

Peters, T., W. L. Adamowicz and P.C. Boxall. 1995. Influence of choice set considerations in modeling 

the benefits from improved water quality. Water Resources Research, Vol (31), pp. 1781–1788, 2000.

Phaneuf, D. 2002. A Random Utility Model for TMDLs: Estimating the Benefits of Watershed Based 

Ambient Water Quality Improvements. Department of Agricultural and Resource Economics, North 

Carolina State University, 2002.

Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality 

Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec 

1998.

Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality 

Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec 

1999.

Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality 

Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec 

2000.

Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality 

Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec 

2001.

Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality 

Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec 

2002.

Phaneuf, Daniel J., Kling, Catherine L., & Herriges, Joseph A. 1998. Valuing Water Quality 

Improvements Using Revealed Preference Methods When Corner Solutions Are Present. 80(5), Dec 

2003.

Phaneuf, Daniel J., Laura O. Taylor, and John Braden. 2013. "Combining RP and SP data to estimate 

preferences for residential amenities: a GMM approach" Land Economics 89(1):30-52.

Roberts, Lisa A., J.A. Leitch. 1997. Economic Valuation of Some Wetland Outputs of Mud Lake. North 

Dakota State University Dept. of Agricultural Economics, Report #381.

Rowe, Robert D., W.D. Schulze, B. Hurd, and D. Orr. 1985. Economic Assessment of Damage Related 

to the Eagle Mine Facility. Energy and Resource Consultants, Inc., Boulder, CO.

Sanders, L. B., R.G. Walsh, and J.B. Loomis. 1990. Toward Empirical Estimation of the Total Value of 

Protecting Rivers. Water Resources Research. Vol. 26, No. 7. pp. 1345 1357.

Schulze, William D., R.D. Rowe, W.S. Breffle, R.R. Boyce, and G.H. McClelland. 1995. Contingent 

Valuation of Natural Resource Damages Due to Injuries to the Upper Clark Fork River Basin. State of 

Montana, Natural Resource Damage Litigation Program. 

Shresta, R. K., J.R.R. Alavalapati. 2004. Valuing Environmental Benefits of Silvopasture Practice: A 

Case Study of the Lake Okeechobee Watershed in Florida. Ecological Economics, Vol (49), pp. 349-359, 

2004.

Smith, V. and W. Desvousges. 1985. The Generalized Travel Cost Model and Water Quality Benefits: A 

Reconsideration. Southern Economic Journal, Vol (52), pp. 371-381, 1985.
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌83

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Stumborg, Basil E., K.A. Baerenklau, and R.C.. Bishop. 2001. Nonpoint Source Pollution and Present 

Values: A Contingent Valuation of Lake Mendota. Review of Agricultural Economics. 23(1): 120-132.

Sutherland, R. J., and R.G. Walsh. 1985. Effect of Distance on the Preservation Value of Water Quality. 

Land Economics. Vol. 61, No. 3. pp. 282 291.

Takatsuka, Yuki, "Comparison of the Contingent Valuation Method and the Stated Choice Model for 

Measuring Benefits of Ecosystem Management: A Case Study of the Clinch River Valley, Tennessee. " 

PhD diss., University of Tennessee, 2004.

Tay, R. McCarthy, P. 1994. Benefits of Improved Water Quality: A Discrete Choice Analysis of 

Freshwater Recreational Demands. Environment and Planning. Vol (26), pp. 1625-1638, 1994.

Trout Unlimited. 2008. An Economic Benefit Analysis for Abandoned Mine Drainage Remediation in the 

West Branch Susquehanna River Watershed, Pennsylvania. Prepared by Downstream Strategies.

Viscusi, Kip W, Joel Huber, and Jason Bell. 2008. The Economic Value of Water Quality. Environmental 

and Resource Economics 41:169 - 187Vollenweider, R.A. 1976. Advances in defining critical loading 

levels for phosphorus lake eutrophication. As cited in Schwarz et al. 2006. Memorie dell’Instituto Italiano 

di Idrobiologia 33:53-83.

Welle, P. G. and Hodgson (2011). "Property Owners’ Willingness To Pay For Restoring Impaired Lakes:

A Survey In Two Watersheds of the Upper Mississippi River Basin." Journal of Applied Business and 

Economics 12(1): 81-94.

Welle, Patrick G. 1986. Potential Economic Impacts of Acid Deposition: A Contingent Valuation Study 

of Minnesota. Dissertation, University of Wisconsin Madison.

Wey, K. A. 1990. Social Welfare Analysis of Congestion and Water Quality of Great Salt Pond, Block 

Island, Rhode Island. Dissertation, University of Rhode Island.

Whitehead, John C., and P.A. Groothuis. 1992. Economic Benefits of Improved Water Quality: a case 

study of North Carolina's Tar Pamlico River. Rivers. 3: 170 178.

Whitehead, John C., G.C. Blomquist, T.J. Hoban, W.B. Clifford. 1995. Assessing the Validity and 

Reliability of Contingent Values: A Comparison of On-Site Users, Off-Site Users, and Non-users. 

Journal of Environmental Economics and Management. Vol. 29.

Whitehead, John C., Hoban, T., and Clifford, W. 2002. Landowners' Willingness to Pay for Water 

Quality Improvements: Jointly Estimating Contingent Valuation and Behavior with Limited Information. 

White paper developed in part by US EPA, NCDENR and the College of Agriculture and Life Sciences at 

NSCU.

Whittington, Dale, G. Cassidy, D. Amaral, E. McClelland, H. Wang and C. Poulos. 1994. The Economic 

Value of Improving the Environmental Quality of Galveston Bay. Department of Environmental 

Sciences and Engineering, University of North Carolina at Chapel Hill.

C.5 : Studies Used in Open Space Valuation Meta-Analysis 

Abt Associates developed a meta-analysis of open space benefits to residential property owners to support 

benefits analysis for the proposed post-construction rulemaking. Abt Associates will update this PQAPP 

with the results of ongoing efforts to update the meta-analysis. Current metadata include: 

Abbott, J. K., & Klaiber, H. A. 2010. Is all space created equal? Uncovering the relationship between 

competing land uses in subdivisions. Ecological Economics, 70(2), 296-307. doi: 

10.1016/j.ecolecon.2010.09.001
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌84

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Acharya, G., & Bennett, L. L. 2001. Valuing Open Space and Land-Use Patterns in Urban Watersheds. 

The Journal of Real Estate Finance and Economics, 22(2-3), 221-237. 

Anderson, S. T., & West, S. E. 2006. Open space, residential property values, and spatial context. 

[Article]. Regional Science and Urban Economics, 36, 773-789. 

Bark, R. H., Osgood, D. E., Colby, B. G., Katz, G., & Stromberg, J. 2009. Habitat preservation and 

restoration: Do homebuyers have preferences for quality habitat? [Article]. Ecological Economics, 68, 

1465-1475. 

Bin, O. 2005. A semiparametric hedonic model for valuing wetlands. Applied Economics Letters, 12(10), 

597-601. doi: 10.1080/13504850500188505

Bin, O., & Polasky, S. 2005. Evidence on the Amenity Value of Wetlands in a Rural Setting. Journal of 

Agricultural and Applied Economics, 37(3), 589-602. 

Bolitzer, B., & Netusil, N. R. 2000. The impact of open spaces on property values in Portland, Oregon. 

Journal of Environmental Management, 59, 185-193. 

Bowman, T., Thompson, J., & Colletti, J. 2009. Valuation of open space and conservation features in 

residential subdivisions. Journal of Environmental Management, 90, 321-330. 

Cho, S.-H., Roberts, R. K., & Kim, S. 2011. Negative externalities on property values resulting from 

water impairment: the case of Pigeon River watershed. Paper presented at the Agricultural & Applied 

Economics Association’s 2011 AAEA & NAREA Joint Annual Meeting, Pittsburgh. 

Cho, S. H., Bowker, J. M., & Park, W. M. 2006. Measuring the Contribution of Water and Green Space 

Amenities to Housing Values: An Application and Comparison of Spatially-weighted Hedonic Models. In 

S.-H. Cho, J. M. Bowker & W. M. Park (Eds.).

Cho, S. H., Clark, C. D., Park, W. M., & Kim, S. G. 2009. Spatial and Temporal Variation in the Housing 

Market Values of Lot Size and Open Space. [Article]. Land Economics, 85, 51-73. 

Cho, S. H., Poudyal, N. C., & Roberts, R. K. 2008. Spatial analysis of the amenity value of green open 

space. Ecological Economics, 66, 403-416. 

Donovan, G. H., & Butry, D. T. 2010. Trees in the city: Valuing street trees in Portland, Oregon. 

Landscape and Urban Planning, 94(2), 77-83. doi: DOI: 10.1016/j.landurbplan.2009.07.019

Doss, C. R., & Taff, S. J. 1996. The Influence of wetland type and wetland proximity on residential 

property values. Journal of Agricultural and Resource Economics, 21(1), 120-129. 

Geoghegan, J., Wainger, L. A., & Bockstael, N. E. 1997. Spatial landscape indices in a hedonic 

framework: an ecological economics analysis using GIS. Ecological Economics, 23, 251-264. 

Hardie, I., Lichtenberg, E., & Nickerson, C. J. 2007. Regulation, open space, and the value of land 

undergoing residential subdivision. Land Economics, 83, 458-474. 

Heintzelman, M. A. 2010. Measuring the property-value effects of local land use and preservation 

referenda. Land Economics, 86, 22-47. 

Irwin, E. G. 2002. The effects of open space on residential property values. Land Economics, 78, 465-

480. 

Kaufman, D., & Cloutier, N. 2006. The Impact of Small Brownfields and Greenspaces on Residential 

Property Values. Journal of Real Estate Finance & Economics, 33(1), 19-30. doi: 10.1007/s11146-006-

8272-7
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌85

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Kopits, E., McConnell, V., & Walls, M. 2007. The trade-off between private lots and public open space in 

subdivisions at the urban-rural fringe. [Proceedings Paper]. American Journal of Agricultural Economics, 

89, 1191-1197. 

Lutzenhiser, M., & Netusil, N. R. 2001. The effect of open spaces on a home's sale price. Contemporary 

Economic Policy, 19(3), 291-298. 

Mahan, B. L., Polasky, S., & Adams, R. M. 2000. Valuing Urban Wetlands: A Property Price Approach. 

Land Economics, 76(1), 100-113. 

Munroe, D. K. 2007. Exploring the determinants of spatial pattern in residential land markets: amenities 

and disamenities in Charlotte, NC, USA. Environment and Planning B-Planning & Design, 34, 336-354. 

Netusil, N. R. 2006. Economic Valuation of Riparian Corridors and Upland Wildlife Habitat in an Urban 

Watershed. Journal of Contemporary Water Research & Education(134), 39-45. 

Netusil, N. R., Chattopadhyay, S., & Kovacs, K. F. 2010. Estimating the demand for tree canopy: a 

second-stage hedonic price analysis in Portland, Oregon. Land Economics, 86(2), 281-293. 

Ready, R., & Abdalla, C. 2003. GIS analysis of land use on the rural-urban fringe: The impact of land use 

and potential local disamenities on residential property values and on the location of residential 

development in Berks County, Pennsylvania Staff Paper 364 (pp. 60). University Park: Pennsylvania 

State University, Department of Agricultural Economics and Rural Sociology.

Sander, H., Polasky, S., & Haight, R. G. 2010. The value of urban tree cover: A hedonic property price 

model in Ramsey and Dakota Counties, Minnesota, USA. Ecological Economics, 69(8), 1646-1656. doi: 

DOI: 10.1016/j.ecolecon.2010.03.011

Sander, H. A., & Polasky, S. 2009. The value of views and open space: Estimates from a hedonic pricing 

model for Ramsey County, Minnesota, USA. Land Use Policy, 26, 837-845. 

Saphores, J.-D., & Li, W. 2012. Estimating the value of urban green areas: A hedonic pricing analysis of 

the single family housing market in Los Angeles, CA. Landscape and Urban Planning, 104(3–4), 373-

387. doi: 10.1016/j.landurbplan.2011.11.012

Shultz, S. D., & King, D. A. 2001. The use of census data for hedonic price estimates of open-space 

amenities and land use. Journal of Real Estate Finance and Economics, 22(2), 239-239. 

Smith, V. K., Poulos, C., & Kim, H. 2002. Treating open space as an urban amenity. Resource and 

Energy Economics, 24, 107-129. 

Stetler, K. M., Venn, T. J., & Calkin, D. E. 2010. The effects of wildfire and environmental amenities on 

property values in northwest Montana, USA. Ecological Economics, 69(11), 2233-2243. doi: DOI: 

10.1016/j.ecolecon.2010.06.009

Thorsnes, P. 2002. The value of a suburban forest preserve: estimates from sales of vacant residential 

building lots. Land Economics, 78(3), 426-441. 

Towe, C. 2009. A valuation of subdivision open space by type. [Proceedings Paper]. American Journal of 

Agricultural Economics, 91(5), 1319-1325. 

Troy, A., & Grove, J. M. 2008. Property values, parks, and crime: A hedonic analysis in Baltimore, MD. 

Landscape and Urban Planning, 87, 233-245. 

White, E. M., & Leefers, L. A. 2007. Influence of natural amenities on residential property values in a 

rural setting. Society and Natural Resources, 20, 659-667. 
Appendix C 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌86

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Williams, E. S., & Wise, W. R. 2009. Economic impacts of alternative approaches to storm-water 

management and land development. Journal of Water Resources Planning and Management-Asce, 135, 

537-546. 
Appendix D 

Abt Associates PQAPP Contract EP-C-13-039 July 2014▌87

Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal.

Appendix D: Telephone Contact Record

Telephone Contact Report

Work Assignment and Task: [Title and number]

Call Date: [Date and Time]

Attendees:

[Full Name, Affiliation]

Topic Discussed/Question Asked (1)

 [Information obtained/Answer to question]

Topic Discussed/Question Asked (2)

 [Information obtained/Answer to question]

Topic Discussed/Question Asked (3)

 [Information obtained/Answer to question]