Environmental Engineering Support 

for Clean Water Regulations 

Programmatic Quality Assurance Project 

Plan (PQAPP) 

Prepared for:

U.S. Environmental Protection Agency 

Engineering and Analysis Division 

Office of Water 

1200 Pennsylvania Avenue, NW 

Washington, D.C. 20460 

Prepared by: 

Eastern Research Group, Inc.

14555 Avion Parkway 

Suite 200 

Chantilly, VA 20151 

October 2013 

Contract No. EP-C-12-021
Revision No. 1 

Date: October 2013 

Page i of viii 

Environmental Engineering Support for Clean Water Regulations 

Programmatic Quality Assurance Project Plan (PQAPP) 

APPROVAL SHEET

Signatures indicate approval of this Programmatic Quality Assurance Project Plan and commitment to 

follow the applicable procedures noted: 

Mary Willett - ERG Contract Quality Assurance Manager Date 

Deb Bartram - ERG Program Manager Date 

Marion Kelly – EAD Quality Assurance Coordinator Date 

Meghan Hessenauer - EPA Technical Project Officer (ERG Support Contract) Date 

Jan Matuszko - EPA Branch Chief, Engineering and Analytical Support Branch Date 

Jan Goodwin - EPA Branch Chief, Technology and Statistics Branch Date 
Revision No. 1 

Date: October 2013 

Page ii of viii 

History of Environmental Engineering Support for Clean Water Regulations 

Programmatic Quality Assurance Project Plan (PQAPP)

Revision 0 

Delivery Date May 15, 2013 

Work Assignment 0-40 

Scope Covers work performed for EAD under Contract EP-C12-021including primary and existing data collection, 

generation, evaluation, analysis, and use. 

Comments Received EPA Comments on First Draft; received 12/6/2013 

EPA Comments on Second Draft: received 3/13/2013 

EPA Comment on Table 10-1: received 5/14/2013 

Comments Addressed ERG Second Draft PQAPP delivered 1/10/2013 

ERG Third Draft PQAPP (without Section 6) delivered 

3/22/2013 

ERG Final PQAPP (without Section 6) delivered via 

FedEx 5/15/2013 

Approval ERG Approvals 5/15/2013 

EPA Approvals 5/29/2013 

Revision 1 

Delivery Date October 28, 2013 

Work Assignment 0-40 

Scope Covers work performed for EAD under Contract EP-C12-021including primary and existing data collection, 

generation, evaluation, analysis, and use. 

Comments Received EPA request to incorporate final version of Section 6 

Comments Addressed ERG PQAPP delivered 10/28/2013 

Approval ERG Approvals 10/28/2013 
Revision No. 1 

Date: October 2013 

Page iii of viii 

TABLE OF CONTENTS 

Page 

1.0 INTRODUCTION .................................................................................................................... 1

2.0 PROJECT MANAGEMENT ELEMENTS .................................................................................... 3

2.1 PQAPP Distribution ............................................................................................................ 3

2.2 Problem Definition/Background ......................................................................................... 4

2.3 Project/Task Description ..................................................................................................... 6

2.3.1 EAD’s Annual 304(m) Review Process ................................................................ 7

2.3.2 Preliminary and Detailed Industry Studies .......................................................... 10

2.3.3 Effluent Limitations Guidelines and Standards Development ............................. 10

2.4 Project Organization ......................................................................................................... 12

2.5 Overall Quality Criteria and Objectives ........................................................................... 20

2.6 Special Training/Certification ........................................................................................... 21

2.7 Quality Assurance Program Documents and Records ...................................................... 22

2.8 Reports to Management .................................................................................................... 22

3.0 COLLECTION OF EXISTING DATA FOR USE IN CONDUCTING EAD’S ANNUAL 304(M)

INDUSTRY REVIEW PROCESS .............................................................................................. 24

3.1 Data Gathering .................................................................................................................. 26

3.1.1 Data Sources ........................................................................................................ 26

3.1.1.1 Data Sources for the TRA and Odd-Numbered Year Analysis ........... 26

3.1.1.2 Additional Data Sources ...................................................................... 27

3.1.2 Criteria for Selecting Data Sources and Resolving Data Gaps ............................ 29

3.1.3 Documentation and Records ................................................................................ 31

3.2 Data Management ............................................................................................................. 31

3.3 Data Quality Assessment .................................................................................................. 32

3.3.1 DMR Pollutant Loading Tool Data Quality Review and Verification ................ 32

3.3.2 Other Existing Data Quality Review and Verification ........................................ 35

3.3.3 Achieving the Contract’s Overall Quality Objectives ......................................... 35

4.0 COLLECTION OF EXISTING DATA FOR USE IN EAD’S INDUSTRY STUDIES AND ELG

DEVELOPMENT................................................................................................................... 38

4.1 Data Gathering .................................................................................................................. 38

4.1.1 Data Needs and Potential Data Sources ............................................................... 38

4.1.2 Literature Searches .............................................................................................. 41

4.1.3 Criteria for Selecting Data Sources and Resolving Data Gaps ............................ 42

4.1.4 Data Value Selection Approach ........................................................................... 43

4.1.5 Data Gathering Documentation and Records ...................................................... 44

4.2 Data Management ............................................................................................................. 44

4.3 Existing Data Quality Assessment .................................................................................... 45

4.3.1 Quality Review of Existing Data Used in Characterizing Industry Point Sources, 

Wastestreams, and Treatment Technologies ........................................................ 45

4.3.2 Quality Review for Existing Data Used in Environmental Assessment Analysis49

4.3.2.1 Qualitative Analyses ............................................................................ 49

4.3.2.2 Quantitative Assessments .................................................................... 50

4.3.3 Data Quality Verification and Reporting ............................................................. 51

4.3.4 Achieving the Contract’s Overall Quality Objectives ......................................... 51
Revision No. 1 

Date: October 2013 

Page iv of viii 

TABLE OF CONTENTS (Continued) 

Page 

5.0 GENERATION OF PRIMARY DATA THROUGH QUESTIONNAIRES AND SURVEYS................... 53

5.1 Data Gathering .................................................................................................................. 53

5.1.1 Questionnaire Data Collection Process Design ................................................... 53

5.1.2 Questionnaire Data Collection Methods .............................................................. 54

5.1.3 Questionnaire Data Handling and Custody .......................................................... 56

5.1.4 Non-Direct Measurements ................................................................................... 56

5.1.5 Site Visit Primary Data Collection ...................................................................... 56

5.2 Questionnaire Data Management ...................................................................................... 57

5.2.1 Transfer of Electronic Questionnaire Responses into a Questionnaire Database 57

5.2.2 Transfer of Paper Questionnaire Responses into a Questionnaire Database ....... 59

5.2.3 Questionnaire Data Documentation ..................................................................... 59

5.2.4 Management of CBI Data in a Questionnaire ...................................................... 60

5.3 Quality Assessment ........................................................................................................... 60

5.3.1 Questionnaire Quality Control ............................................................................. 60

5.3.2 Questionnaire Data Acceptance Criteria .............................................................. 63

5.3.3 Site Visit Primary Data Review and Acceptance Criteria ................................... 64

5.3.4 Achieving the Contract’s Overall Quality Objectives ......................................... 65

6.0 GENERATION OF PRIMARY DATA THROUGH SAMPLING AND ANALYSIS ............................ 66

6.1 Field Sampling .................................................................................................................. 67

6.1.1 Sampling Process Design ..................................................................................... 67

6.1.2 Sampling Methods ............................................................................................... 68

6.1.2.1 Inspection/Acceptance Criteria for Supplies, Consumables, and 

Sampling Equipment ........................................................................... 69

6.1.2.2 Sampling Equipment Calibration, Maintenance, and Decontamination

 ............................................................................................................. 70

6.1.2.3 Methods for Minimizing Sample Contamination ................................ 71

6.1.2.4 Methods for Identifying Upset Conditions Requiring Suspension of 

Sample Collection ............................................................................... 71

6.1.3 Sample Handling and Custody ............................................................................ 71

6.1.4 Field Data Management ....................................................................................... 73

6.1.4.1 Direct Measurements ........................................................................... 73

6.1.4.2 Non-Direct Measurements ................................................................... 73

6.1.5 Field Quality Control Measures and Goals .......................................................... 74

6.1.5.1 Trip Blank ............................................................................................ 75

6.1.5.2 Bottle Blank ......................................................................................... 75

6.1.5.3 Field Blank .......................................................................................... 76

6.1.5.4 Equipment Blank ................................................................................. 76

6.1.5.5 Field Duplicates ................................................................................... 76

6.1.6 Sampling Program Data Quality Criteria and Goals ............................................ 77

6.1.6.1 Completeness ....................................................................................... 79

6.1.6.2 Sensitivity ............................................................................................ 79

6.1.6.3 Precision .............................................................................................. 79

6.1.6.4 Representativeness .............................................................................. 80

6.1.6.5 Sample Contamination ........................................................................ 80

6.1.6.6 Comparability ...................................................................................... 80
Revision No. 1 

Date: October 2013 

Page v of viii 

TABLE OF CONTENTS (Continued) 

Page 

6.2 Requirements for Laboratories Participating in EAD Sampling Programs ...................... 80

6.3 Laboratory Analysis .......................................................................................................... 81

6.3.1 Laboratory Sample Handling and Custody .......................................................... 82

6.3.2 Analytical Methods .............................................................................................. 82

6.3.3 Instrument/Equipment Calibration and Frequency .............................................. 83

6.3.4 Analytical Quality Control ................................................................................... 83

6.3.4.1 Method Detection Limits (MDLs) ....................................................... 83

6.3.4.2 Ongoing Precision and Recovery (OPR) Standard .............................. 84

6.3.4.3 Calibration Check Standard ................................................................. 84

6.3.4.4 Laboratory Blanks ............................................................................... 84

6.3.4.5 Matrix Spike/Matrix Spike Duplicate/Laboratory Duplicate .............. 85

6.3.5 Preparation and Analytical Batches ..................................................................... 86

6.3.6 Internal Standards, Labeled Compounds, Surrogates .......................................... 86

6.3.7 Non-Standard EPA Wastewater Methods ............................................................ 86

6.3.8 Analytical Data Management .............................................................................. 87

6.3.8.1 Analytical Data Review ....................................................................... 87

6.3.8.2 Analytical Data Reduction .................................................................. 89

6.3.9 Analytical Data Reporting ................................................................................... 90

6.4 Achieving the Contract’s Overall Quality Objectives ...................................................... 91

7.0 GENERATION, EVALUATION, AND/OR ANALYSIS OF DATA USING SPREADSHEETS ............ 92

7.1 Developing Spreadsheets .................................................................................................. 92

7.2 Quality Control and Data Quality Goal ............................................................................ 92

7.3 Documenting Spreadsheet Quality Control ...................................................................... 93

7.4 Identification and Control of CBI in Spreadsheets ........................................................... 93

7.5 Quality Assurance ............................................................................................................. 93

8.0 GENERATION, EVALUATION, AND/OR ANALYSIS OF DATA USING DATABASES ................. 94

8.1 Developing Databases ....................................................................................................... 94

8.2 Developing Database Queries ........................................................................................... 94

8.3 Quality Control and Data Quality Goal ............................................................................ 95

8.3.1 QC for Database Data Entry ................................................................................ 95

8.3.2 QC for Database Queries and Outputs ................................................................. 96

8.4 Documenting Database Quality Control ........................................................................... 96

8.5 Identification and Control of CBI in Databases ................................................................ 96

8.6 Quality Assurance ............................................................................................................. 96

9.0 GENERATION, EVALUATION, AND/OR ANALYSIS OF DATA USING MODELS ....................... 97

9.1 ERG Generated Computational Models ........................................................................... 97

9.2 Existing Models ................................................................................................................ 98

9.2.1 Review of Data Quality in Existing Models ........................................................ 99

9.2.2 Review of Model Methodology ........................................................................... 99

9.3 Additional Resources ...................................................................................................... 100

10.0 DELIVERABLES REVIEW ................................................................................................... 101

10.1 Levels of Review ............................................................................................................ 101
Revision No. 1 

Date: October 2013 

Page vi of viii 

TABLE OF CONTENTS (Continued) 

Page 

10.2 Types of Review ............................................................................................................. 103

10.2.1 Technical Review .............................................................................................. 103

10.2.2 Copyediting Review .......................................................................................... 103

10.2.3 Technical Edit Review ....................................................................................... 104

10.3 Quality Assurance for Deliverable Review .................................................................... 104

11.0 ERG’S QUALITY PROGRAM ASSESSMENTS AND RESPONSE ACTIONS .............................. 105

11.1 ERG Quality Program Assessments ............................................................................... 105

11.2 Response Actions ............................................................................................................ 106

12.0 ASSESSING DATA USABILITY AND RECONCILIATION WITH USER REQUIREMENTS ........... 107

12.1 Assessment of Data Usability ......................................................................................... 107

12.2 Reconciliation with User Requirements ......................................................................... 107

13.0 REFERENCES .................................................................................................................... 109

Attachment A: PUBLICLY-OWNED TREATMENT WORKS (POTW) PERCENT REMOVALS 

USED FOR THE TRIRELEASES2002 DATABASE

Attachment B: COMMENTS RECEIVED REGARDING POTW REMOVALS 

Attachment C: TELEPHONE CONTACT REPORT 

Attachment D: 304(m) PROJECT FILE MANAGEMENT PLAN 

Attachment E: QUESTIONNAIRE MAILOUT CHECKLIST 

Attachment F: DATA REVIEW GUIDELINES 

Attachment G: METHOD SPECIFIC CHECKLIST 

Attachment H: DELIVERABLE REVIEW SIGN-OFF SHEET 

Attachment I: QA AUDIT CHECKLIST 
Revision No. 1 

Date: October 2013 

Page vii of viii 

LIST OF TABLES 

Page 

2-1 PQAPP Distribution List......................................................................................................3

2-2 Work Activities and Typical Key Questions by Program ...................................................4

2-3 Contract EP-C-12-021 EPA Project Staff Responsibilities ...............................................15

2-4 Contract EP-C-12-021 ERG Project Staff Responsibilities ...............................................16

3-1 Additional Data Sources Identified for the Even-Numbered Annual Reviews .................27

3-2 304(m) Annual Review Data Sources Acceptance Criteria ...............................................36

4-1 Existing Data Purposes and Data Elements Gathered .......................................................39

4-2 Existing Data Acceptance Criteria for Industry Studies and ELG Development ..............47

5-1 Industry Questionnaire Data Acceptance Criteria .............................................................64

6-1 Inspection and Acceptance Criteria for Supplies, Consumables, and Sampling 

Equipment ..........................................................................................................................69

6-2 Field Quality Control Samples...........................................................................................74

6-3 Field Sampling and Analytical Data Quality Review Criteria ...........................................77

6-4 Data Quality Qualifiers for Sampling Program .................................................................88

6-5 Typical Analytical Database Structure ..............................................................................90

10-1 Levels of Review .............................................................................................................102
Revision No. 1 

Date: October 2013 

Page viii of viii 

LIST OF FIGURES 

Page 

2-1 Integration of the 304(m) Annual Review Process with EAD Studies and ELG 

Development ..................................................................................................................6

2-2 304(m) Annual Review Process .....................................................................................9

2-3 Effluent Guidelines Development Process ..................................................................11

2-4 ERG Program Management Structure for Contract EP-C-12-021 ..............................14
Revision No. 1 

Date: October 2013 

Section No. 1 

Page 1 of 109 

A Supplemental QAPP provides 

project/work assignment-specifics 

and/or procedures that are not covered 

by the PQAPP or that have been 

added or modified into the work. 

1.0 INTRODUCTION

The scope of this Programmatic Quality Assurance Project Plan (PQAPP) is Eastern Research Group, 

Inc.’s (ERG’s) Environmental Engineering Support for Clean Water Regulations Contract (Engineering 

Contract) EP-C-12-021 and encompasses the activities included in the contract Performance Work 

Statement (PWS). This PQAPP covers the collection, 

generation, evaluation, analysis, and use of environmental 

data. ERG intends this PQAPP to be used as is or in concert 

with a Supplemental QAPP (SQAPP) to explain ERG’s 

quality assurance and quality control (QA/QC) procedures 

for conducting work under this contract. This PQAPP covers 

the following types of programs that ERG supports for the 

U.S. Environmental Protection Agency (EPA) Engineering and Analysis Division (EAD): 

 304(m) annual review process; 

 Preliminary and detailed studies of specific industries and point source categories; and 

 Development and implementation of effluent limitation guidelines and standards (ELGs). 

In support of these programs, ERG conducts the following types of activities that will require QA/QC: 

 Collection and analysis of existing data for use in conducting EAD’s annual 304(m) industry 

review process; 

 Collection and analysis of existing data for use in EAD’s industry studies and ELG development; 

 Collection/generation and evaluation of primary data through questionnaires or surveys 

(including written and/or telephone surveys that may or may not require an Information 

Collection Request [ICR] and primary data collected during site visits); 

 Collection/generation and evaluation of primary data through sampling and analysis; 

 Generation, evaluation, and analysis of data using spreadsheets; 

 Generation, evaluation, and analysis of data using databases; 

 Generation, evaluation, and analysis of data from ERG designed cost and loading models, 

existing Agency models, and ERG-designed or off-the-shelf models; and 

 Preparation of draft and final deliverables. 

In preparing work assignments to be issued by EAD under ERG’s Contract EP-C-12-021, the EPA Work 

Assignment Manager (WAM) and EAD Quality Assurance Coordinator will review the work assignment 

activities against the activities covered by this PQAPP and any existing project-specific QAPPs. The EPA 

WAM will determine what work assignment activities will be covered under this PQAPP, what activities 

will require a SQAPP to be developed, and whether a separate project-specific QAPP will be required. 

These requirements will be specified in the work assignment issued by EAD. 

Offices and divisions other than EAD that issue work under ERG’s Contract EP-C-12-021 should 

evaluate their QA/QC needs and determine if this PQAPP fulfills part or all of those needs. The offices 
Revision No. 1 

Date: October 2013 

Section No. 1 

Page 2 of 109 

and divisions will specify in each work assignment whether the work is covered under this PQAPP, an 

SQAPP is required, or a project-specific QAPP is required. 

This document is organized as follows: 

 Section 2.0 presents Project Management elements that are generic to all ERG EAD projects and 

work assignments conducted under this contract; 

 Sections 3.0 – 10.0 present activity-specific data gathering, data management, data quality 

assessment, and data verification and validation elements; 

 Section 11.0 discusses ERG’s internal quality program assessments and response actions that 

apply to EAD projects/work assignments conducted under this contract; and 

 Section 12.0 discusses ERG’s approach for assessing data usability and reconciliation with user 

requirements for EAD projects/work assignments conducted under this contract. 

EAD’s effluent guidelines work is subject to unplanned deviations as projects/work assignments progress. 

This PQAPP acknowledges the likelihood of deviations over time. ERG will update this document, 

SQAPPs, and project-specific QAPPs to reflect changes as appropriate. ERG staff is responsible for 

implementing this PQAPP’s QA/QC procedures under this contract. EPA’s WAMs are responsible for 

specifying the QA/QC documentation required for any public record or docket created under this contract. 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 3 of 109 

2.0 PROJECT MANAGEMENT ELEMENTS

2.1 PQAPP Distribution

The PQAPP distribution list will include the staff identified in Table 2-1: 

Table 2-1. PQAPP Distribution List 

Name 

Title Contact Information Mailing Address 

U.S. Environmental Protection Agency, Engineering and Analysis Division 

Meghan Hessenauer 

Technical Project Officer 

(202) 566-1040 

Hessenauer.Meghan@epa.gov 

U.S. Environmental Protection Agency 

Office of Science and Technology 

1200 Pennsylvania Avenue, NW (4303T)

Washington, DC 20460 

(202) 566-1053 (fax) 

Marion Kelly 

EAD Quality Assurance 

Coordinator 

(202) 566-1045 

Kelly.Marion@epa.gov 

Jan Matuszko 

Branch Chief, Engineering and 

Analytical Support Branch 

(202) 566-1035 

Matuszko.Jan@epa.gov 

Jan Goodwin 

Branch Chief, Technology and 

Statistics Branch 

(202) 566-1060 

Goodwin.Janet@epa.gov 

EAD Work Assignment Manager Specified in the EPA Work 

Assignment 

Eastern Research Group, Inc. 

Deborah Bartram 

ERG Program Manager 

(703) 633-1669 

Deborah.Bartram@erg.com 

Eastern Research Group, Inc. 

14555 Avion Parkway 

Suite 200 

Chantilly, VA 20151-1102 

(703) 263-7280 (fax) 

Mary Willett 

ERG Contract Quality Assurance 

Manager 

(703) 633-1615 

Mary.Willett@erg.com 

ERG Work Assignment Manager Specified in ERG Work Plan 

ERG Work Assignment Project 

Team1 Specified in ERG Work Plan 

 

1

 The ERG project team will include the Project QA Coordinator and may include subcontractor staff. If 

subcontractor staff are used to support a work assignment, the ERG WAM will provide the subcontractor manager, 

QA Manager (if applicable), and technical staff with applicable portions of this PQAPP and will review appropriate 

QA/QC procedures at the start of the work. 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 4 of 109 

2.2 Problem Definition/Background

Each EAD work assignment, work assignment amendment, and/or technical directive issued under the 

contract will include details about the background of the technical work and will define the activities 

required by the project. Table 2-2 presents the typical work activities that ERG will support and the key 

questions that define the problems addressed in each EAD program that will be conducted under the 

contract. 

Table 2-2. Work Activities and Typical Key Questions by Program 

Program Work Activities 

Typical Key Questions to be Answered by 

the Project Work Statement

304(m) Annual Review ERG’s analysis of existing data sources 

will provide technical support for EPA 

preparation of Effluent Guidelines 

Program Plans required by CWA Section 

304(m). ERG will provide technical 

support for the following activities: 

 Annual review of effluent guidelines 

and pretreatment standards that EPA 

has promulgated under the Clean 

Water Act (CWA); 

 Identification of categories of sources 

discharging toxic or nonconventional 

pollutants for which guidelines or 

pretreatment standards have not 

previously been published; 

 Establishing a schedule for further 

investigation and possible 

development or revision of effluent 

guidelines and pretreatment standards; 

 Preparation of technical documents; 

 Preparation of regulatory rulemaking 

records and public dockets; and

 Evaluation of public comments. 

To prepare the Final Effluent Guidelines 

Program Plans, EAD will seek to answer, 

among others, the following questions: 

 What industrial wastewater discharges 

pose the greatest potential threat to 

human health or the environment? 

 What industrial wastewater discharges 

should be studied further? 

 What industrial wastewater discharges 

warrant possible ELG development or 

revision? 

EAD Industry Studies 

and/or ELG Development 

ERG’s collection of primary and existing 

data will support EPA’s technical 

understanding of industry profiles, 

industry processes that generate 

wastewaters subject to the CWA, 

pollutant discharges and their impacts, 

pollutant controls, and the costs and 

benefits of pollutant controls. Primary and 

existing data are also typically used to 

support environmental data evaluation 

and analysis. 

ERG will provide technical support for 

the following types of activities: 

 Engineering site visits; 

In collecting primary and existing data as part 

of an industry study or effluent guideline 

development project, EAD seeks to answer 

the following questions: 

 What facilities are included in a specified 

industry category? Where are they 

located? What is their size? 

 What are the primary sources of 

wastewater within an industry category? 

 What are the characteristics of the raw 

wastewaters (i.e., pollutants, 

concentrations, flow rate/volume)? 

 What pollutants are discharged in the final 

effluent of an industry category operation? 

 What pollutant discharges require 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 5 of 109 

Table 2-2. Work Activities and Typical Key Questions by Program 

Program Work Activities 

Typical Key Questions to be Answered by 

the Project Work Statement

 Field sampling activities; 

 Questionnaire development and 

implementation; 

 Technical literature reviews; 

 Industry profile development; 

 Technology options development; 

 Technology performance assessments; 

 Engineering cost estimation; 

 Evaluation of regulatory alternatives; 

 Preparation of technical documents; 

 Preparation of regulatory rulemaking 

records and public dockets; 

 Evaluation of public comments; and 

 Rulemaking implementation support.

additional control? What are the pollutants 

of concern? 

 What are the concentrations and loadings 

of pollutants currently being discharged 

(i.e., “baseline” concentration)? 

 What are the environmental impacts 

resulting from discharge of pollutants of 

concern? What is the toxicity associated 

with the pollutants of concern? 

 What are the best available technologies 

for pollution prevention and wastewater 

treatment that reduce discharges of 

pollutants of concern and to what levels? 

 How will industry discharges change if 

plants control pollutants using these 

technologies/practices? 

 What are the capital and operating costs of 

the best available technologies/practices? 

 What are the environmental impacts 

associated with baseline conditions? What 

are the environmental impacts associated 

with various regulatory options? 

Figure 2-1 presents the integration of the 304(m) annual review process with EAD’s selection of industry 

categories for studies and ELG development. The 304(m) annual review informs EAD’s decisions on 

which industry categories to evaluate further and which categories will be addressed through an ELG. 

Based on the decisions made by EAD as part of the 304(m) annual review, ERG may be directed to 

support a preliminary or detailed study and/or support an ELG development program. ERG’s support to a 

study and/or ELG development includes collection of existing and primary data, evaluation and analysis 

of those data to assess wastewater discharges, pollutants, discharge loads, and applicable treatment and/or 

pollution prevention technologies to reduce pollutant loads, documentation of analysis through draft and 

final deliverables, preparation of rulemaking records, and review and evaluation of public comments. 

ERG also develops calculation tools and models to assess industry-specific costs and load removals that 

result from application of regulatory options and environmental impacts associated with baseline and 

post-regulation discharges. 

ERG’s 304(m), preliminary or detailed study, and ELG support is conducted using an interactive process 

with EPA. During data collection, ERG will assess information collected and/or identified, will discuss its 

quality and usefulness with the EPA WAM and, based on those discussions, may conduct additional data 

collection or may proceed with data analysis support. During ERG’s data analysis support activities, 

analysis results are checked as described in this PQAPP and reviewed with the EPA WAM. Based on this 

collaborative discussion of analysis and evaluation, EPA will direct ERG to collect additional data, 

perform additional analysis, conduct additional evaluation, and/or prepare documentation on the analysis 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 6 of 109 

results. While ERG may recommend use of specific data or approaches, the EPA WAM will be 

responsible for deciding data utility. 

Figure 2-1. Integration of the 304(m) Annual Review Process with EAD Studies and ELG 

Development 

2.3 Project/Task Description

This section presents general program-level descriptions of activities related to the 304(m) annual review 

process, industry studies, and ELG development projects. EPA issues individual work assignments to 

ERG for specific projects with project-specific task descriptions. 2 EPA’s work assignments will specify 

which PQAPP sections apply to the work to be conducted under the work assignment. As necessary, ERG 

will supplement this PQAPP with project-specific SQAPPs, study plans, and sampling and analysis plans 

to identify the type(s) of data that will be gathered or generated, and where and how the data will be 

obtained. The programs supported under the EAD contract are described below in terms of program 

purpose and types of activities conducted. For each of these programs, ERG will generate project-related 

documentation and deliverables as part of the project file. As needed, ERG uses project file 

documentation and EPA-directed deliverables to populate a public record. Also, for each of these 

programs ERG may use spreadsheets, databases, and/or modeling to conduct the work. Sections 7.0, 8.0, 

and 9.0 of this PQAPP present details on ERG’s QA/QC activities while using Microsoft Excel® 

spreadsheet, Microsoft Access® database, and modeling tools, respectively, as part of our data evaluation 

and analysis efforts. Section 10.0 presents details on ERG’s QA/QC activities associated with 

development and finalization of written project deliverables. 

 

2

 Projects supporting the development of effluent guidelines may span multiple work assignments. In addition, some 

work assignments, such as those supporting EPA’s annual review and detailed industry studies, may comprise 

multiple projects. 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 7 of 109 

EPA Contract EP-C-12-021 has a contract period of performance of September 26, 2012, through 

September 25, 2017. All work will be performed during this period of performance and in accordance 

with the timeframes specified in individual work assignments issued by EPA. 

2.3.1 EAD’s Annual 304(m) Review Process 

The 1972 CWA directs EPA to develop national regulations placing limits on the pollutants that are 

discharged by categories of industry to rivers and streams (termed “effluent guidelines”) or to sewage 

treatment plants3

 (termed “pretreatment standards”). The Act also directs EPA to develop national 

regulations for new industrial facilities (termed “new source performance standards”). An additional 

critical component of the Act is that it requires EPA to periodically study and review existing effluent 

guidelines, pretreatment standards, and new source performance standards and to consider the need to 

develop regulations for industries not covered by a national regulation. These planning requirements are 

found in several sections of the CWA. Section 304(m) provides for an effluent guideline plan that 

contains three basic elements to be published on February 4, 1987, and biennially thereafter. 

First, EPA must establish a schedule for the annual review of existing effluent guidelines promulgated 

under Section 304(b) (i.e., limitations for existing direct dischargers) and for annual revision of the 

guidelines if appropriate (see Section 304(m)(1)(A)). Second, EPA must identify categories of sources 

that directly discharge toxic or non-conventional pollutants for which EPA has not published effluent 

limitations guidelines or new source performance standards (see Section 304(m)(1)(B)). Third, EPA must 

establish a schedule for the establishment of national regulations for any categories identified in the 

second step, with a promulgated regulation three years after identification in a national plan (see Section 

304(m)(1)(C)). 

For indirect dischargers, Section 304(g) requires EPA to review at least annually and, if appropriate, 

revise the pretreatment standards EPA has promulgated under CWA Section 307. In addition, Section 

307(b) provides that EPA must promulgate pretreatment standards for categories of sources not subject to 

existing pretreatment standards if there is pass-through or interference at POTWs. As a good government 

practice, EPA publishes the findings of its annual reviews of direct and indirect dischargers together in 

one document, the “Effluent Guidelines Program Plan” (ELG Plan). Typically, EPA publishes a 

preliminary Plan in odd-numbered years and publishes a final Plan in even-numbered years after public 

review on the preliminary Plan. 

To support EPA’s annual review process, ERG collects existing data from EPA’s National Pollutant 

Discharge Elimination System (NPDES), the Toxics Release Inventory (TRI) program, and other existing 

data sources as described in Section 3.1.1 to help address the questions identified in Table 2-2. ERG and 

EPA’s evaluation of these data support assessment of which facilities and industry categories are 

discharging the largest amounts of toxic weighted pollutants. For those industries that discharge 

significant toxic pollutants that are not already subject to effluent limitations and guidelines, EPA 

assesses whether additional study of the industry is warranted. ERG also evaluates whether there have 

been significant changes within industry categories subject to existing ELGs that might require revision of 

the regulations. 

 

3

Also referred to as publicly owned treatment works (POTWs).
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 8 of 109 

Figure 2-2 presents the basic steps of EPA’s annual review process for odd- and even-numbered years. 

ERG supports EPA by supplementing the industry categories of interest identified through the evaluation 

of NPDES and TRI data with other sources of existing data to identify new industry categories that need 

to be assessed. 

Based on the annual review process, EPA will prioritize for further review (i.e., a detailed study or 

preliminary category review) those industrial categories whose pollutant discharges potentially pose the 

greatest hazards to human health or the environment because of their toxicity (i.e., categories that 

collectively discharge over 95 percent of the total toxic weighted pounds). EPA also considers efficiency 

and implementation issues raised by stakeholders in identifying candidates for further review. By using 

this layered approach, the Agency concentrates its resources on those point source categories with the 

highest estimates of toxic-weighted pollutant discharges (based on best available data), while assigning 

lower priority to categories that the Agency believes are not good candidates for ELG revision. 

Table 2-2 presents a listing of the work activities ERG performs in support of the 304(m) annual review 

process. Section 3.0 presents details on the existing data sources evaluated for the 304(m) annual review 

process and the QA/QC activities conducted by ERG when supporting that program. 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 9 of 109 

Figure 2-2. 304(m) Annual Review Process 

 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 10 of 109 

2.3.2 Preliminary and Detailed Industry Studies 

For those industrial point source categories identified through the 304(m) annual review process for 

additional study, EAD initiates a preliminary or detailed study to collect additional data. EPA uses the 

data from these studies to assess whether the Agency should initiate an ELG rulemaking activity to either 

establish new or revise existing regulations on wastewater discharges from that category. ERG’s data 

collection and analysis process for an industry study is similar to that conducted under a full ELG 

development project; however, the industry study typically entails a more limited data collection and 

analysis effort. To support EPA’s industry studies, ERG collects publicly available information on the 

universe of facilities in the point source category, wastewaters generated by the industry’s processes, 

existing characteristics of those wastewaters (e.g., pollutants, concentrations, flows), and preliminary 

information on the types of treatment and pollution prevention processes used by the industry (or 

applicable to the wastewaters generated). Using the information ERG collects, EAD will assess and 

determine whether development of a full ELG is warranted. 

Table 2-2 presents a listing of the work activities ERG performs in support of industry studies. Section 

4.0 presents details on the existing data sources used to support preliminary and detailed industry studies 

and the QA/QC activities conducted by ERG when performing these studies. 

2.3.3 Effluent Limitations Guidelines and Standards Development 

For those industrial point source categories identified for ELG development, EAD conducts a series of 

data collection, evaluation, and analysis steps to identify technology-based regulatory options and 

calculate associated regulatory limits and standards to control pollutant discharges from the point source. 

Figure 2-3 presents the basic steps of the ELG process. ERG’s support to EAD in this process includes 

collection of existing and primary data, evaluation and analysis of these data, and presentation of the 

analyses in draft and final deliverables that document the data and EAD’s decision making process using 

the data. ERG supports most of the steps outlined in Figure 2-3 with the exception of designating 

industries for regulation (Step 1 of Figure 2-3) which is performed by EPA, analysis of economic impacts 

and environmental benefits (Step 7, Step 12, Step 13c, and Step 13(d) of Figure 2-3) which is conducted 

under a separate contract, calculation of effluent limits (Step 15 of Figure 2-3) which is done by EPA’s 

statistical group, and selection and publication of the preferred option (Step 16 of Figure 2-3) which is 

performed by EPA. ERG supports development of industry costs for various regulatory options which are 

then used as part of the economic impact analysis and development of loads for various regulatory options 

which are then used as part of the environmental benefits analysis. ERG may also support environmental 

analysis and assessments of selected point source categories under study. ERG supports proposal and 

final regulatory efforts by providing regulatory record materials generated as part of the rule to EPA’s 

Federal Document Management System. 

Table 2-2 presents a listing of the work activities ERG performs in support of ELG development. Sections 

4.0, 5.0, and 6.0 of this PQAPP present details on existing and primary data collection activities 

performed by ERG to support the ELG process and the QA/QC activities conducted by ERG. 

 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 11 of 109 

Figure 2-3. Effluent Guidelines Development Process 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 12 of 109 

2.4 Project Organization

ERG’s program management structure for Contract EP-C-12-021 is shown in Figure 2-4. Each ERG 

project under this contract, issued to ERG through individual work assignments, will be directed by the 

EPA WAM designated in the work assignment. ERG’s Engineering Contract Program Manager, Deb 

Bartram will assign an ERG WAM who will coordinate the work assignment team at ERG. Deb Bartram 

will provide technical oversight of all work completed on the contract and will work in coordination with 

the EPA Project Officer, Meghan Hessenauer and QA staff. As ERG’s Contract QA Manager, Mary 

Willett will work with Deb Bartram to implement the quality requirements that EPA has specified for this 

contract. Ms. Willett will also interact with the EAD Quality Assurance Coordinator, Marion Kelly who 

will work with each of the EPA WAM’s in determining the PQAPP quality requirements that apply to 

each individual work assignment. Most work assignments/projects under the contract will include an 

ERG Project QA Coordinator who will work with the project team and the project technical reviewers to 

ensure that ERG’s QA/QC systems are used. The ERG WAM’s will have overall responsibility to ensure 

that the project team adheres to the requirements of the PQAPP (or other supplemental and/or projectspecific QAPPs) and they will work with the ERG Project QA Coordinator to support and assess 

compliance. In addition, ERG has a corporate quality staff made up of a Corporate QA Manager and 

Local QA Coordinators in each of our main offices. These quality staff act as a resource to all ERG staff 

in understanding ERG’s quality system and support projects in setting up quality procedures and tools. 

Work plans developed for each work assignment under the contract will name those ERG individuals 

fulfilling the roles identified in this figure. 

If subcontractors are used on a work assignment, ERG’s WAM will interact with the subcontractor WA 

Manager and any subcontractor QA Manager. ERG’s WAM will provide the subcontractor staff with a 

copy of applicable portions of this PQAPP and will ensure that the QA/QC requirements of the work 

assignment are understood by the subcontractor staff at the start of the work. Any specific quality 

requirements will be included in the subcontract. The ERG WAM is responsible for ensuring that 

subcontractor products meet the requirements of this PQAPP. The ERG Program Manager has ultimate 

responsibility for ensuring subcontractor compliance with ERG’s PQAPP (or other supplemental and/or 

project-specific QAPPs). If ERG identifies any quality issues while working with a subcontractor, ERG’s 

Program and Work Assignment Managers and the subcontractor’s Program and Work Assignment 

Managers will become involved and participate in identifying corrective action. 

The responsibilities of EPA project staff by project role are summarized below in Table 2-3. The 

responsibilities of ERG project staff by project role are summarized below in Table 2-4. 

ERG technical staff supporting work under the work assignments of Contract EP-C-12-021 will have 

various project support roles as specified by ERG’s work plans. In addition, the ERG WAMs for 

sampling projects will assign a project staff member to act as Crew Chief for each sampling episode 

conducted. ERG’s sampling Crew Chiefs have overall responsibility for ERG’s field work and manage 

and coordinate sampling activities as described in Section 6.0 below. For projects that involve developing 

and utilizing spreadsheets, the ERG WAM will designate a spreadsheet developer. The spreadsheet 

developer is responsible for creating and using the spreadsheet and ensuring that it meets project needs as 

described in Section 7.0 below. The ERG WAMs for work assignments involving database development 

will assign an ERG project staff member(s) to act as database task leader and/or developer and query 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 13 of 109 

developer. ERG’s database task leaders are responsible for coordinating the design and development of 

databases to meet project needs as described in Section 8.0 below. 
Figure 2-4. ERG Program Management Structure for Contract EP-C-12-021 
Table 2-3. Contract EP-C-12-021 EPA Project Staff Responsibilities 

Project Role Summary of Responsibilities

EPA Project Officer The EPA Project Officer is responsible for defining overall quality objectives and overall direction on the 

duties of the EPA Project Officer include: 

 Participating in meetings with ERG managers, EPA WAMs, the EAD QA Coordinator and other EPA s

collection needs and quality assurance matters under the contract. 

 Reviewing major deliverables under the contract, as appropriate.  Assisting EPA WAMs, EPA Managers, and staff from other organizations within and outside EPA, in m

the data gathered under the contract. 

 Working with the EPA QA Officer and QA Coordinators, ERG QA Manager, and ERG managers to ide

necessary to ensure that quality objectives under the contract are met.

EPA WAM Each EPA WAM will report to the EPA PO regarding activities assigned to ERG under individual work as

WAMs are responsible for ensuring that the policies described in the PQAPP are supplemented with work 

Sampling and Analysis Plans (SAPs), and other planning documents that detail any additional procedures ncollection activities performed under their work assignments meet work assignment needs. EPA WAMs arfollowing: 

 Participating with senior EPA program staff and technical personnel, and as appropriate ERG technical 

planning for projects to be conducted under their work assignment, including the development of projecmeasures of quality, and acceptance criteria. 

 Identify the resources needed for each project to be performed under their work assignment, including q

needs, and requesting funding from the Branch Chief to meet those needs. 

 Identifying and implementing project-specific quality management procedures, which may include data 

information management, data integration, and data validation. 

 Completing the quality Assurance Review Form for each work assignment involving environmental dat

quality system requirements. 

 Ensuring that work assignments, work plans, and contract deliverables include quality system document

activity. 

 Reporting any deviations from the QAPPs used under this contract to the EAD Quality Assurance Coor Preparing and implementing quality system documentation appropriate for the project.  Monitoring contractor performance against the work plan and quality system documentation (e.g., this P Review the performance of ERG staff.  Communicating with and directing the ERG WAM regarding requirements and schedules for tasks to be 
Table 2-3. Contract EP-C-12-021 EPA Project Staff Responsibilities 

Project Role Summary of Responsibilities

work assignment. 

EAD QA Coordinator The EAD QA Coordinator reports to the EAD Director and is responsible for: 

 Reviewing and approving this PQAPP and any other SQAPPs or project-specific QAPPs developed und Monitoring whether all QA procedures described in the PQAPP and any other SQAPPs or project-speci

under the contract are followed. 

 Assisting the EPA WAMs in implementing corrective actions necessary to resolve any deviations from 

Table 2-4. Contract EP-C-12-021 ERG Project Staff Responsibilities 

Project Role Summary of Responsibilities

ERG Program Manager The ERG Program Manager organizes and directs the technical activities and the contract and functions as 

between the client, management, and the project team. The duties of the Program Manager include: 

 Communicating with the EPA client on technical matters.  Informing project staff of contract requirements and ensuring that the staff follows the requirements.  Reviewing contract modifications, work assignments, work plans, and cost estimates.  Ensuring that the project is assigned the appropriate ERG staffing levels subcontractors, and technical e Initiating and reviewing subcontractor authorization to work.  Managing and reviewing technical and financial progress reports and invoices.  Implementing ERG’s Security Plan for Handling Confidential Business Information under the Clean W

12-021. 

 Implementing ERG’s quality management plan (QMP) as specified in this PQAPP and any supplementa

QAPPs prepared to support work assignments issued under the contract. 

 Ensuring that ERG remains free of conflicts of interest.  Working with the Project QA Coordinator and the ERG WAMs to resolve any quality problems that ari Serving as the point of contact with ERG management on all matters relating to the contract.  Coordinating the technical components (including personnel, facilities, and equipment) required under t Monitoring the technical activities on projects under the contract to ensure that the technical objectives, 
Table 2-4. Contract EP-C-12-021 ERG Project Staff Responsibilities 

Project Role Summary of Responsibilities

met. 

 Reviewing performance of ERG WAMs, subcontractors, and consultants.

ERG WAM ERG WAMs coordinate the activities of the individual work assignments performed under the contract. In 

ERG WAMs: 

 Are available to the EPA client for action on any problem related to specific work assignments.  Responsible for designating ERG staff for preparing any required SQAPPs or project-specific QAPPs.  Prepare and implement work plans, test plans, health and safety plans, and associated project instruction

assignments and responsible for following the requirements of this PQAPP and/or supplemental QAPP as specified by the EPA WAM. 

 Provide the EPA client with periodic status briefings or reports.  Implement ERG’s cost and performance tracking and control system.  Complete projects on time, within budget, and in accordance with the work order or assignment technic

objectives. 

 Maintain and document all project-related records in individual project files (these may include files, ca

and professional engineering judgments). 

 Follow ERG’s PQAPP which complies with ERG’s QMP, to ensure the technical quality of reports, me

communications from inception to delivery. 

 Monitor the technical activities of each ERG work assignment staff member to ensure that all staff are m

technical standards and adhering to the budget and schedule. 

 Review the performance of ERG staff.  Keep the ERG Program Manager informed on all aspects of each task, including expenditures, technical 

recommended solutions. 

 Monitor subcontractor performance and provide performance data to the ERG Program Manager.  Ensure compliance with all QC acceptance criteria as specified in any SQAPP or project-specific QAPP Keep the ERG QA staff and the ERG Program Manager advised of any quality problems that arise.  Discuss any significant quality deficiencies with the EPA WAM and ensure that actions to correct quali

documented, and reported to the ERG Project QA Coordinator. 

ERG Contract QA The ERG Contract QA Manager is responsible for: 
Table 2-4. Contract EP-C-12-021 ERG Project Staff Responsibilities 

Project Role Summary of Responsibilities

Manager  Working with the ERG Program Manager to develop this PQAPP and ensure that contract-specific QA/

commensurate with contract objectives and with ERG’s quality system. 

 Assisting the Program Manager and ERG WAMs in identifying and assigning appropriate Project QA C

reviewers. 

 Reviewing with Project QA Coordinators the QA/QC procedures developed and executed for each work 

with the PQAPP, SQAPP or individual QAPPs. 

 Conducting independent audits to determine the effectiveness of the Environmental Engineering Suppor

program. 

 Issuing stop work orders for quality considerations.  Maintaining records of contract internal QA audits. 

ERG Project QA Coordinator 

Project QA Coordinators are responsible for: 

 Implementing this PQAPP and/or supporting preparation of a project-specific QAPP/SQAPP that docum

assignment-specific policies, organization, objectives, functional activities, and specific QA/QC procedu

quality goals or requirements as required by the work assignment. This includes reviewing work-assignm

 Verifying that the requirements of the approved PQAPP and other ERG QA/QC procedures are commu

assignment staff. 

 Serving as internal consultants to the Program Manager and ERG WAMs in defining quality goals or re

developing a work-assignment-specific QC system that is responsive to them. 

 In consultation with the project management, verifying that subcontractors’ and consultants’ quality-rela

adequate and executed. 

 Bringing quality problems to the immediate attention of the ERG Program Manager, including a recomm

replace staff, or other changes as needed or, if warranted, to the attention of the Contract QA Manager f

corrective action. 

 Providing an independent assessment of performance through QA audits.  Overseeing any external QA audit activities requested by the client if required.  Documenting the results of all QA/QC activities in reports to ERG management and, if required, to the 

ERG Technical Reviewer Technical reviewers review project deliverables: 

 To determine if the deliverable objectives have been fulfilled in a technically sound manner. 
Table 2-4. Contract EP-C-12-021 ERG Project Staff Responsibilities 

Project Role Summary of Responsibilities

 To ensure that the deliverable is clearly written.  To check that calculations are correct and references are fully documented. 

ERG Sampling Crew Chief 

The ERG sampling crew chief has overall responsibility for ERG’s field work on a specific sampling epis

 Coordinating development of the Sampling Plan and Health and Safety Plan for a specific sampling epis Managing and coordinating sampling activities as described in Section 6.0 of this PQAPP. 

ERG Spreadsheet Developer 

The ERG spreadsheet developer is responsible for: 

 Creating, naming, and developing the spreadsheet.  Ensuring that data in the spreadsheet are imported or entered correctly, clearly identified and labeled, pr

mathematical functions used for manipulating data are executed accurately as described in Section 7.0. 

ERG Database Task Lead/Developer 

The ERG database task lead/developer is responsible for: 

 Coordinating the design and development of databases and/or developing databases to meet project need Ensuring that database QC procedures are followed and documented as described in Section 8.0. 

ERG Project Staff Project staff are responsible for executing the work assignment QA/QC requirements specified in this PQA

QAPPs/SQAPPs, and other work assignment plans. They are responsible for documenting the results of th

for communicating with the Project QA Coordinator and ERG WAM regarding quality issues and concern
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 20 of 109 

2.5 Overall Quality Criteria and Objectives

The overall quality needed to support EAD’s 304(m) annual review process, preliminary and detailed 

studies, and ELG development and implementation projects is discussed below. Sections 3.0 through 6.0 

present activity-specific data quality criteria and associated data quality specifications and acceptance 

criteria that apply to ERG’s support of EAD’s programs. 

EAD uses EPA’s Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity 

of Information Disseminated by the Environmental Protection Agency (EPA, 2002) as the basis for 

determining the quality of data needed to support this contract. These Information Quality Guidelines 

(IQGs) are designed to ensure the Agency’s compliance with the Data Quality Act (also known as the 

Information Quality Act), and address the following goals: 

Disseminated information should adhere to a basic standard of quality, including objectivity, utility, and 

integrity; 

The principles of information quality should be integrated into each step of EPA’s development of 

information, including creation, maintenance, and dissemination; and 

Administrative mechanisms for correction should be flexible, appropriate to the nature and timeliness of 

the disseminated information and incorporated into EPA’s information resources management practices. 

As stated in the EPA IQGs, the Agency “works every day to ensure information quality, but does not wait 

until the point of dissemination to consider important quality principles...” The IQGs further define 

quality based on the following three characteristics: 

 Objectivity: (1) The information must be accurate, reliable, and unbiased, and (2) the manner in 

which the information is presented must be accurate, clear, complete, and unbiased; 

 Integrity: The information may not be compromised through corruption or falsification, either by 

accident, or by unauthorized access or revision, and 

 Utility: The information must be useful for the intended users. 

Within EAD, these IQGs are implemented in accordance with Office of Water Information Quality 

Guidelines: Pre-dissemination Review Guidance and Checklists, version 2.3, September 23, 2010. The 

EPA and OW IQGs distinguish between influential information, influential scientific risk assessment 

information, and non-influential information. Based on these requirements, EAD has determined that data 

will be collected and used on this contract to support influential scientific information. Accordingly, the 

data collection, manipulation, and analyses performed under this contract must meet the objectivity, 

integrity, and utility data quality objectives defined above, and must provide “transparency regarding (1) 

the source of the data used, (2) the various assumptions employed, (3) the analytical methods applied, and 

(4) the statistical procedures employed.” 

ERG discusses these overall quality objectives in the context of the data quality procedures of this 

PQAPP in Sections 3.3.3, 4.3.4, 5.3.4, and 6.4 below. 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 21 of 109 

2.6 Special Training/Certification

ERG has the following in-house training programs that apply to work conducted for EAD under this 

contract. 

CWA Confidential Business Information (CBI) Security Training/Certification – ERG maintains an 

approved CWA CBI Security Plan. The plan is reviewed annually by any staff that will access or use 

CWA CBI, and ERG conducts certification for CBI use via a test and authorization form. The ERG 

Program Manager (Deb Bartram) is responsible for identifying any additional program staff that will need 

to access CWA CBI and to inform them and the Document Control Officer (DCO) of the need for staff 

training. ERG’s DCO or Deputy DCO will issue the plan and test staff as required. 

Health and Safety (H&S) Program – ERG has a corporate H&S program that includes a medical 

monitoring program and a minimum of 24 hours H&S training for staff prior to engineering site visits or 

sampling participation. ERG conducts an annual 8-hour refresher H&S training for any Chantilly, VA 

office staff who are expected to conduct site visits and/or sampling episodes. The ERG Program Manager 

is responsible for identifying any ERG staff anticipated for site or sampling activities and they are 

included in ERG’s H&S program. ERG’s Chantilly, VA office H&S coordinator is Steve Strackbein, who 

plans and coordinates our annual Chantilly, VA office H&S refresher training and additional H&S 

training as needed. 

Sampling Training - ERG has previously conducted and will continue to periodically conduct an inhouse sample collection training session that reviews wastewater sample collection, preservation, 

packaging, and shipping procedures specific to ERG’s Office of Water support. As needed, ERG will also 

conduct training on clean hands/dirty hands sampling techniques specific to the collection of wastewater 

samples for low-level metals analyses. The ERG Program Manager and appropriate ERG WAMs will 

identify staff who are anticipated to be deployed for sampling and will determine whether and what type 

of sampling training is needed and at what frequency. Generally, ERG will plan to conduct sampling 

training prior to anticipated sampling programs for EPA’s Office of Water and will ensure that any new 

staff who has not participated in the training attend. 

Hazardous Materials Shipping Training – Part of ERG’s annual H&S refresher training includes a 

review of International Air Transport Association’s (IATA’s) hazardous materials shipping requirements. 

Quality System Training – ERG conducts training on ERG’s corporate quality policy and systems for all 

technical staff on an annual basis. ERG’s Corporate QA Manager (Mary Willett) and the Local QA 

Coordinators are responsible for coordinating and conducting this training. 

Training and certification requirements beyond those listed above and identified as necessary as part of a 

specific project/work assignment will be discussed in ERG’s work plans (or SQAPP) along with our 

process for conducting/obtaining the necessary training or obtaining the necessary certification. 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 22 of 109 

2.7 Quality Assurance Program Documents and Records

Quality assurance program documentation for the EP-C-12-021 contract includes: 

 ERG’s customized QMP included in our contract and modified under WA 0-40 to incorporate 

additional information provided in the QMP Addendum; 

 This contract-level PQAPP, which is fully compliant with ERG’s customized QMP; 

 Contract-related paperwork, including the project-specific work assignments, work assignment 

amendments, and technical directives (see Section 2.2) and ERG work plans (see Section 2.4); 

 Where specified, SQAPPs and/or project-specific QAPPs (see Section 1.0); 

 ERG Standard Operating Procedures (SOPs) referenced in ERG’s customized QMP and 

discussed in Sections 6.0 (sampling procedures), 7.0 (spreadsheet quality control procedures), 8.0 

(database quality control procedures), 9.0 (model development and use quality control 

procedures), and 10.0 (deliverable review); 

 ERG’s planning documents, including Sampling and Analysis Plans (see Section 6.1.1), Study 

Plans (see Section 2.3.2), and systematic planning process memoranda (see Sections 5.1.1 and 

8.1); and 

 Project files that discuss and document QA/QC-related activities (see Sections 3.1.3, 3.2, 4.2, 

5.1.3, 5.2.3, 6.1.4, and 6.3.8). 

Discussions of these project specific documentation and records occur throughout this document. ERG’s 

QAPPs are “living” documents and subject to modification as the work becomes more fully defined. To 

maintain version control, each QAPP has a header identifying the revision number of the PQAPP, 

SQAPP, or other QAPPs and subsequent revisions will include a front table identifying significant 

changes from the previous version. At the direction of the EAD Quality Assurance Coordinator, ERG will 

update this PQAPP resulting in a subsequent version. Alternatively, modifications or additions to the 

QA/QC procedures conducted to support work under this contract may be covered by a SQAPP or a 

project/work assignment-specific QAPP. EPA’s WAMs will direct the need for SQAPPs or projectspecific QAPPs. 

2.8 Reports to Management

When directed to do so by the EPA WAMs, ERG will develop task-specific quality review documentation 

to present a summary of data reviews against stated data quality objectives (DQOs) and/or data quality 

goals. ERG will provide any required data quality documentation to the EPA WAM and will include it in 

the project files. Details on quality documentation by type of activity are described in subsections 3.3.2, 

4.3.3, 5.3.1, 6.1.6, 6.3.8.1, 7.3, and 8.4 below. Additionally, ERG will provide each EPA WAM with 

monthly reports on the status of QA activities. These reports will be prepared by the ERG WAM and 

incorporated into our monthly technical progress reports (TPRs), which will be supplemented to include 

the following: 

 Quality assurance activities performed during the reporting period as part of the implementation 

of the work assignment; 
Revision No. 1 

Date: October 2013 

Section No. 2 

Page 23 of 109 

 Identification of any problems encountered; 

 Identification of any deviations from the applicable QAPPs (PQAPP, SQAPP, or project-specific 

QAPP); and 

 Identification of problem resolution and/or corrective actions taken during the reporting period, if 

any. 

Minor errors identified during the quality control checks described in this PQAPP will be addressed 

through the normal preparation and review process for a given deliverable. Significant quality 

deficiencies (pervasive errors or quality issues that could impact project budget and schedule) detected by 

technical reviewers or the ERG Project QA Coordinator will be communicated, in writing, to the ERG 

WAM (with copies to the ERG Program Manager and ERG Contract QA Manager) for resolution. The 

ERG WAM will discuss significant quality deficiencies with the EPA WAM and is responsible for 

ensuring that action to correct the quality deficiencies is taken, documented, and reported to the ERG 

Project QA Coordinator. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 24 of 109 

Toxic Weighted Pound Equivalents 

(TWPE) is calculated as the pounds 

of pollutant times the pollutant’s toxic 

weighting factor. Information about 

EPA’s Office of Water toxic 

weighting factors is available at: 

water.epa.gov/lawsregs/lawsguidance/

cwa/304m/upload/2005_09_19_guide

_304m_2006_toxic_weighting.pdf

and will be available in the 304(m) 

annual review records. 

3.0 COLLECTION OF EXISTING DATA FOR USE IN CONDUCTING EAD’S ANNUAL 304(M)

INDUSTRY REVIEW PROCESS

In support of the 304(m) annual review, ERG conducts environmental data collection, evaluation, and 

analysis activities on an alternating year schedule. EPA uses an odd- and even-numbered year approach to 

review industrial discharges according to the CWA, publishing a draft plan (for public comment) in oddnumbered years and a final plan in even-numbered years. Each year, ERG composes an Annual Review 

Report (ARR), which provides details on how data were generated, any data quality checks and 

corrections made, and the findings from analyses. Under this contract, ERG will continue to conduct oddand even-numbered year analyses and will prepare future AARs. 

During odd-numbered years, ERG will conduct a toxicity rankings analysis (TRA) of all industrial 

discharges, including categories subject to existing ELGs and discharges for industries with no categorical 

regulation. ERG will use the results from the TRA to prioritize the categories for further review based on 

toxicity of pollutant discharges ranked highest to lowest. To generate the TRA, ERG will use the EPA 

Office of Compliance’s DMR Pollutant Loading Tool4

, available at http://cfpub.epa.gov/dmr/, which 

provides category rankings by Toxic Weighted Pound 

Equivalents (TWPE). The DMR Pollutant Loading Tool uses 

discharge monitoring report (DMR) data and TRI data 

providing category rankings that can be drilled down to the 

facility and pollutant level. 

The TRA focuses on identifying the “top industrial 

dischargers” of pollutants in effluent wastewater, expressed 

as TWPE. Top industrial dischargers are generally defined as 

industries that collectively produce 95 percent of discharges 

in the United States, as measured in units of TWPE. As 

discussed in Section 3.3.1, ERG will conduct a review of the 

data for the top industrial dischargers, to identify errors and 

anomalies. After discussing a list of likely errors and anomalies with the EPA WAM, ERG will conduct 

follow-up calls as directed by the EPA WAM to specific facilities to verify the data inputs and obtain any 

necessary corrections. 

ERG will prepare a table documenting the resulting database changes, which will reference 

teleconferences and other backup data, such as permits, verifying the validity of the database change. 

ERG will include this table in the ARR, so that all data corrections can be viewed by the public. ERG will 

work with EPA’s Office of Compliance to make corrections to the underlying DMR Pollutant Loading 

Tool data so that future versions of the tool will include the updated, corrected data. ERG will continue 

these iterations of identifying data errors and corrections and re-ranking industrial discharges by TWPE, 

as directed by the WAM. 

From the corrected data, and at the EPA WAM’s direction, ERG will perform further review of the 

discharges in industry categories with higher TWPE. ERG will investigate these industry sectors, 

including specific facilities, to understand the process source of the discharge, the pollutants in the 

 

4

 DMR Pollutant Loading Tool Version 1.1 was released in November 2012. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 25 of 109 

discharge, pollutant mass and pollutant concentrations. Using additional record information, including 

following up with facilities and/or permit writers, engineering textbooks, scientific journal articles, and 

public comments on the previous EPA Preliminary or Final ELG Plans, ERG will support EPA in 

characterizing the source(s) of the pollutant discharges. ERG will also identify and analyze categories or 

sectors with significant changes from the previous year (e.g., order of magnitude changes either higher or 

lower) in factors such as discharge amounts, changes in unit processes, advances in analytical method 

techniques, and pollution prevention advances or innovative technology usage or availability. In 

particular, ERG will review categories according to the following considerations: 

 Amount of toxic pollutants discharged, whether large amounts of pollutants are discharged, 

and/or potential hazard to human health or the environment; 

 Number of discharging facilities; 

 Whether a demonstrated technology, process change, or pollution prevention approach could be 

used to control pollutants; 

 Cost, performance, and affordability of the technology, process change, or pollution prevention 

measure; and 

 Implementation and efficiency considerations. 

The study of these categories or sectors will primarily focus on: 

 Verification of TRI- and DMR-reported discharges; 

 Preliminary identification of process source(s) of discharged pollutants; and 

 Limited identification of potential control alternatives. 

ERG will summarize its odd-numbered year review and prioritized categories in the ARR. 

During even-numbered years, ERG will use additional data sources to supplement the TRA and identify 

unregulated industrial discharges not captured by the DMR and TRI data. Such data sources are described 

in Section 3.1.1.2. The even-numbered year reviews will also seek to identify and review wastewater 

discharges that are not quantified in DMR/TRI data or not prioritized based on TWPE, such as nutrients, 

contaminants of emerging concern, and conventional pollutants. Also, where more data or analyses are 

needed, as directed by the EPA WAM ERG will continue its reviews of categories identified during oddnumbered year reviews. ERG will support EPA’s end product of the even-numbered year review, the final 

ELG Plan, which publishes the results of the annual review and EPA’s schedule for any effluent 

guidelines development. 

ERG will also perform additional in-depth “preliminary” or “detailed” studies or a somewhat less detailed 

“preliminary category review” in even-numbered years, as directed by the EPA WAM. In this way, EPA 

identifies existing categories for potential ELG revision or new categories for ELG development. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 26 of 109 

3.1 Data Gathering

To answer the key questions identified in Table 2-2 in support of the preliminary and final ELG Plans, 

EPA needs to gather data characterizing industrial wastewater discharges and related treatment residues 

and data identifying the sources of the wastewater. 

3.1.1 Data Sources 

This section provides general descriptions of the data sources that ERG will use for the 304(m) annual 

reviews. Quality criteria and specifications for the collected data are discussed in Section 3.3. ERG will 

obtain the existing data from databases, publications, other EPA offices, www.regulations.gov, and other 

websites. 

3.1.1.1 Data Sources for the TRA and Odd-Numbered Year Analysis 

For the TRA and odd-numbered year analysis, ERG will use the following data sources: 

 DMR Data. ERG will use DMR data reported for major dischargers and, where available, for 

minor dischargers. ERG will use permit facility data, parameter limits data, and 

measurement/violation data for the year 2011 (or later as available). The DMR Pollutant Loading 

Tool provides a report entitled Technical Users Background Document for the Discharge 

Monitoring Report (DMR) Pollutant Loading Tool, which describes the origin and maintenance 

of these datasets (available at 

http://cfpub.epa.gov/dmr/docs/Technical_Users_Background_Doc.pdf). 

 TRI Data. ERG will use TRI data from the public release of reporting year 2011 (or later as 

available) to analyze TRI-reported releases to surface waters and POTWs. ERG will use data on 

chemical releases to water and off-site transfers to POTWs, the basis for release and transfer 

estimates, Chemical Abstracts Service (CAS) numbers, facility location and contact information, 

wastewater treatment information, dioxin distribution information, and releases to other media for 

select facilities. 

 POTW Removal Percentages. ERG will use data from recently-developed pretreatment standards 

to identify chemical-specific removal rates typical of POTWs nationwide. The memorandum 

entitled “Publicly-Owned Treatment Works (POTW) Percent Removals Used for the 

TRIReleases2002 Database” (Codding, 2005) describes the data sources used to identify POTW 

removal rates used in the database (see Attachment A). The memorandum entitled “Comments 

Received Regarding POTW Removals” (Bicknell, 2006) describes the comments received and 

the changes made to the POTW removals (see Attachment B). ERG will use the POTW removal 

rates in these memoranda (or subsequent updates to these memoranda) to estimate the quantity of 

chemicals transferred to POTWs that are ultimately discharged to receiving streams. 

In addition, ERG will also reference the following as needed: 

 Point Source Category Crosswalk. ERG will rely on the Standard Industrial Classification (SIC) / 

North American Industry Classification System (NAICS) code to Point Source Category 

Crosswalk embedded in the DMR Pollutant Loading Tool. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 27 of 109 

 CAS and Pollutant Parameter Groupings. For the point source category rankings, ERG will rely 

on the pollutant parameter groupings built into the DMR Pollutant Loading Tool. ERG would use 

the ungrouped parameter loadings only for facility-specific reviews or further review of specific 

industries. For example, ERG might evaluate if a facility reported dissolved metals instead of 

total metals when considering accuracy of loads or evaluating treatment technologies. 

3.1.1.2 Additional Data Sources 

At the direction of the EPA 304(m) project WAM, ERG identified data sources other than the DMR 

Pollutant Loading Tool to expand EPA’s ability to identify new pollutants of concern or wastewater 

streams in industrial categories not currently regulated by ELGs. ERG will use these additional data 

sources (listed in Table 3-1) to research pollutants not included in the TRA and also to identify improved 

wastewater treatment technologies. 

Since 2003, EPA has relied on the TRA to support the annual reviews and planning process. EPA has also 

considered other data submitted to the Agency through the public review process. The TRA provides a 

national view of pollutant loadings discharged to surface water, identifies industries with the largest 

discharges, and highlights data errors in the underlying databases. However, this TRA process has 

consistently identified the same few industrial categories with high TWPE. The TRA’s usefulness in 

identifying categories is decreasing because it provides: 

 Data for industries and pollutants that are already regulated under NPDES permits or that are 

required to report to TRI. Therefore, new industries, waste streams, and/or pollutants are difficult 

to identify with the TRA. 

 No information on the state of the pollutant control technology used to achieve such discharges; 

therefore, it is difficult to determine if pollutant discharges could be further reduced from current 

levels. 

 No information on the economics of the industries discharging pollutants, which could be used to 

evaluate if they could afford more stringent controls. 

Because of these limitations with the TRA, ERG identified, starting in 2012, additional data sources for 

the even-numbered annual reviews. ERG chose these sources seeking to identify new pollutants, pollution 

control technologies or alternatives, or new industries of concern not already categorized or covered by 

the TRA. Table 3-1 lists the data sources identified and recommended by ERG for their potential 

relevance to the annual reviews. 

Table 3-1. Additional Data Sources Identified for the Even-Numbered Annual Reviews 

Data Source Relevant Content Data of Interest 

Office of Air Quality 

Planning and Standards 

(OAQPS) air regulations: 

New Source Review (NSR), 

National Emission Standards 

for Hazardous Air Pollutants 

(NESHAPs), and New Source 

Air regulations may require air pollution controls 

that generate new wastewater streams for industrial 

categories. 

Treatment technology 

descriptions; performance data; 

technology costs. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 28 of 109 

Table 3-1. Additional Data Sources Identified for the Even-Numbered Annual Reviews 

Data Source Relevant Content Data of Interest 

Performance Standards 

(NSPS) 

Office of Water Targeted 

National Sewage Sludge 

Survey (TNSSS) 

Sewage sludge survey showing concentrations of 

pollutants in sewage sludge may link to industrial 

wastewater discharges. 

Discharge data, pollutants in 

sewage sludge, pollutant mass, 

pollutant concentration, flows. 

Water Environment 

Federation technical journals 

Technical journal articles and presentations may 

identify wastewater treatment advances. 

Treatment technology 

descriptions, performance data, 

technology costs, sector trends 

(new facilities, processes, 

feedstocks). 

Office for Chemical Safety 

and Pollution Prevention/ 

Office of Pollution 

Prevention and Toxic 

Substances (OCSPP/OPPTS), 

Significant New Use Rules, 

Chemical Data Reporting 

Rule (CDRR), and Chemical 

Action Plans 

Health and environmental hazard data for 

chemicals manufactured in or imported into the 

United States, which may or may not be regulated 

by ELGs. Also, new chemical manufacturing 

processes with unregulated discharges of new 

pollutants. 

Process descriptions, pollutants 

in discharges, environmental 

effects data, qualitative 

descriptions of impacts, human 

health effects data. 

Office of Research and 

Development (ORD), 

National Risk Management 

Research Laboratory and 

New Analytical Methods 

Recent analytical methods that identify new 

pollutants of concern that are not already regulated 

by the ELGs (e.g., perfluorinated compounds and 

other contaminants of emerging concern). 

Pollutants that can be identified 

by the method. 

2011 TRI Expansion Sectors 

(or subsequent updates) 

Supporting data for industrial sectors reviewed as 

part of the expansion may indicate pollutant 

discharges of concern. 

Pollutants in discharges, 

pollutant mass, pollutant 

concentration, flows. 

Targeted stakeholder input in 

response to EPA outreach to 

selected industry groups 

EPA Regions, state and local regulatory agencies, 

industry, pretreatment coordinators, and citizen 

groups often collect data related to ELGs and 

industrial discharges outside of public comment. 

Pollutants in discharges, 

pollutant mass, pollutant 

concentration, flows, process 

descriptions, treatment 

technology descriptions, 

performance data, technology 

costs, sector trends (new 

facilities, processes, feedstocks). 

Office of Groundwater and 

Drinking Water (OGWDW) 

OGWDW priorities that identify industries and 

contaminants of emerging concern, such as 

hydraulic fracturing, and emerging treatment 

technologies for drinking water that can be used to 

treat wastewater (identified in OGWDW’s 

Treatability Database). 

Treatment technology 

descriptions, process 

descriptions, performance data, 

technology costs. 

U.S. Geological Survey 

(USGS), data warehouses and 

Publications on topics of emerging concern and 

USGS data warehouses that focus on ambient 

water quality but provide a potential means of 

Pollutants in discharges, 

impaired water descriptions. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 29 of 109 

Table 3-1. Additional Data Sources Identified for the Even-Numbered Annual Reviews 

Data Source Relevant Content Data of Interest 

publications prioritizing impaired waters and/or pollutants 

causing impairments. 

Assessment and Watershed 

Protection Division, impaired 

waters data 

Impaired waters data to identify potential 

pollutants of concern. 

Pollutants in discharges, 

impaired water descriptions, 

pollutants in impaired waters. 

Office of Water Standards 

and Health Protection 

Division, water quality 

criteria 

Water quality criteria currently under development 

to determine if any industries directly relate to a 

specific pollutant discharge. 

Water quality criteria. 

Office of Enforcement and 

Compliance Assurance 

(OECA), enforcement 

initiatives 

Compliance initiatives may show new pollutants of 

concern or waste streams in industries that warrant 

review or revision. 

Pollutant in discharges, pollutant 

mass, sector trends, 

documentation of environmental 

impacts. 

Public comments on the EPA 

304(m) plans 

Comments on draft and final ELG plans often 

provide data. 

Pollutants in discharges, 

pollutant mass, pollutant 

concentration, flows, process 

descriptions, treatment 

technology descriptions, 

performance data, technology 

costs, sector trends (new 

facilities, processes, feedstocks). 

3.1.2 Criteria for Selecting Data Sources and Resolving Data Gaps 

EPA prioritized the review of data sources in Table 3-1 in 2012 and will consider reviewing other data 

sources in future years. EPA has prioritized reviewing data sources based on (1) likelihood of identifying 

unregulated discharges, and (2) utility in identifying new wastewater treatment technologies or pollution 

prevention alternatives. 

 Likelihood of Identifying Unregulated Industrial Discharges. Some of EPA’s program offices 

focus on contaminants of emerging concern that are not currently characterized by DMR and TRI 

data, such as the supporting documentation for OCSPP’s Chemical Action Plans for higher-chain 

perfluorinated compounds and the 2011 TRI Expansion Sector effort. 

 Utility in identifying new wastewater treatment technologies or pollution prevention alternatives.

For specific industries, EPA collects data to characterize the performance and cost of applicable 

and demonstrated wastewater treatment technologies, process changes, or pollution prevention 

alternatives that could effectively reduce pollutant discharges during preliminary category 

reviews and detailed studies. For the 2012 review, EPA selected data sources that report on 

treatment technologies and pollution prevention alternatives across all industries. EPA’s reviews 

in future years may instead focus on selected technologies or pollution prevention practices. This 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 30 of 109 

PQAPP will be revised or a SQAPP will be developed to incorporate this focus when, and if, it 

occurs. 

For the 2012 annual review, EPA selected the first seven data sources on Table 3-1for ERG review. EPA 

will follow a similar rationale for future even-numbered years, identifying a prioritized list of data sources 

in the odd-numbered year report that will receive public comment. Based on the odd-numbered year 

proposed list of sources, the EPA WAM will direct ERG on which sources to review and ERG will 

document the future data source selection in the ARR. ERG’s qualitative analyses will rely on these 

sources as well as published literature identified during review of materials related to the targeted 

analyses. The criteria that ERG will use to evaluate the quality of literature are: 1) accuracy, 2) reliability, 

and 3) representativeness. 

 Accuracy. For purposes of this PQAPP, ERG assumes that the underlying data and information 

contained in state and federal reports and peer-reviewed journal articles are accurate. ERG 

assumes that data from these sources are of sufficient quality to identify potential environmental 

impacts associated with industrial discharges. ERG’s WAM will present data that are outliers 

(e.g., data that are not consistent with the body of information collected from the literature 

review) to the EPA WAM during their weekly status calls and will receive further direction on 

whether to incorporate or exclude the data in the analysis. Section 3.3.2 discusses data quality 

reviews for these data sources. 

 Reliability. During the review of existing literature, ERG may encounter sources that contain 

limited documentation of the environmental impact claim (e.g., unidentified speaker at a 

stakeholder meeting or a local resident commenting in a newspaper article). These data sources 

represent a lower level of reliability than peer-reviewed literature, but may still provide useful 

information on the range of environmental impacts associated with industrial discharges. If ERG 

identifies limitations in the quality of information taken from other existing data sources (e.g., 

websites, newspapers, and stakeholder meetings), the ERG WAM will communicate these data 

quality issues to the EPA WAM during their weekly status calls. ERG will also evaluate data 

reliability based on whether: 

- Scientific work is clearly written, so that all assumptions and methodologies can be 

identified; 

- Variability and uncertainty (quantitative and qualitative) in the information or in the 

procedures, measures, methods, or models are evaluated and characterized; and 

- Assumptions and methodologies are consistently applied throughout the analysis as reported 

in the source. 

 Representativeness. EPA requires data that have a national perspective and are representative of 

an industry. ERG will evaluate representativeness of existing data based on whether they 

represent: 

- Types of environmental impacts caused by wastewater discharges; 

- Range of concentrations; and 

- Environments affected (e.g., rivers, lakes, and wetlands). 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 31 of 109 

ERG will use the criteria described in Section 3.3.2 below to identify if collected data are usable for this 

program. ERG will also evaluate the reasonableness of existing data. For example, pollutants must be 

reasonably related to the process or waste stream from which they were discharged. ERG may flag 

datasets as questionable and subject to further investigation if a chemical is detected that is not expected 

in the waste stream or if an expected chemical is missing (based on historical data from earlier annual 

reviews). ERG will communicate to the EPA WAM whether reported or calculated pollutant loads, plant 

wastewater flows, or other existing data reflect the range of expected values, again based on historical 

data. In addition, if the data provided by a data source are repeatedly found to be unreasonable, ERG will 

discontinue use of that data source and communicate these quality issues to EPA. 

ERG will not review the quality of existing data contained in published EPA databases and data systems 

(e.g., TNSSS or other databases provided by federal or state agencies or trade organizations but will 

describe known limitations of the data in reports on which they are based. If ERG identifies limitations in 

the quality of the data from published literature or federal/state generated databases that may impact 

decisions regarding the overall direction of further data collection efforts or the quality of deliverables, 

the ERG WAM will communicate these issues to the EPA WAM during their weekly status calls. Data 

limitation discussions will identify where data gaps exist and describe data reporting requirements. 

3.1.3 Documentation and Records 

ERG staff will use the following procedures when collecting published data and placing it in the project 

file: 

 Collect full bibliographic information on all collected publications, including “date accessed” for 

information collected from the Internet; 

 Save web pages as PDF files, so that the data source can be added to the electronic project file; 

and 

 Use the following procedures when collecting data via telephone and email contact: 

- Document all telephone contacts using a Telephone Contact Report depicted in Attachment 

C; 

- Save all email contacts as PDF files, so that they can be added to the electronic project file; 

and 

- Ensure that telephone contact reports and email records are added to the electronic project 

file. 

Data gathered will be documented by the ERG project staff as source files in the project folder, which is 

maintained electronically on ERG’s project server. ERG’s WAM in consultation with the EPA WAM will 

determine which materials gathered during the 304(m) annual review process need to be included in the 

record that supports the ARR. 

3.2 Data Management

ERG’s 304(m) Project File Management Plan, Revised June 2009 (ERG, 2009) is provided in 

Attachment D and presents detailed procedures that will be used for managing project files assembled in 

support of EPA’s preliminary and final ELG Plans and procedures for submitting appropriate portions of 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 32 of 109 

these files to the docket. The ERG WAM assigns a Project Assistant the responsibility of maintaining the 

file index and filing documents in the centralized project file (including the electronic project file). The 

ERG WAM also ensures that project team members document all data sources used. 

ERG also follows requirements for identification and control of different versions of the DMR Pollutant 

Loading Tool and any spreadsheet and/or database tables created to present and summarize collected data. 

These requirements include: 

 Assigning a unique name and number (date or version number) to spreadsheets and/or databases 

created ; 

 Maintaining a log that identifies spreadsheet and/or database revisions and describes the changes 

made to the different versions (log may be a database table, a separate electronic file in the same 

folder as the database, or a file stored on removable media); and 

 Maintaining a historical file of the DMR Pollutant Loading Tool or spreadsheet/database used to 

support the annual review. 

Section 7.0 presents details on ERG’s management of data in spreadsheets and Section 8.0 presents 

details on ERG’s management of data in project databases. 

3.3 Data Quality Assessment

The primary objectives of the 304(m) annual data reviews are to ensure that: 

 Data generated and collected are of the quality necessary for EPA to answer the questions 

identified in Table 2-2; and 

Data analysis results are documented in the 304(m) project file, and analyses are of the type, quality, 

transparency, and reproducibility needed. 

Descriptions of data quality criteria and review procedures that ERG will use to identify data that are fit 

for this use are described in this section. ERG will collect information about industrial wastewater 

discharges from existing data sources. ERG anticipates collecting data on: 

 Pollutant loads discharged by specified industries and/or facilities; 

 Wastewater characteristics (e.g., volume, mass loads, pollutant concentrations); 

 Industrial sector products and pollution control trends; 

 Wastewater treatment technology performance/efficiency; 

 Wastewater treatment technology costs; and 

 Environmental and human health effects of industrial wastewater discharges. 

3.3.1 DMR Pollutant Loading Tool Data Quality Review and Verification 

The review and evaluation of data pulled from the DMR Pollutant Loading Tool includes quality checks 

that occur within the loading tool programming and quality checks where ERG runs specific queries using 

the loading tool and/or examines accompanying data, such as permit documents. 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 33 of 109 

EPA evaluates the quality of the DMR data for use in developing the current year’s loading database (e.g., 

DMR data for 2011) as part of the biennial TRA. This evaluation considers data completeness, accuracy, 

reasonableness, and comparability for facilities and industry categories ranked in the top 95th percentile, 

based on TWPE. The Technical Users Background Document for the Discharge Monitoring Report 

(DMR) Pollutant Loading Tool (EPA, 2012) documents the built-in quality checks, which include the 

load calculator routines. (This document is available at 

htpp://cfpub.epa.gov/dmr/docs/Technical_Users_Background_Doc.pdf). 

The following discussion describes additional quality review steps that ERG will perform at the EPA 

WAM’s direction: 

 Completeness, reasonableness, and comparability of DMR and TRI data. Using data in the DMR 

Pollutant Loading Tool, ERG will compare DMR and TRI data from different years to evaluate 

its completeness, reasonableness, and comparability. For example, as part of the 2012 304(m) 

annual review, ERG will use the DMR Pollutant Loading Tool to compare the number of 

facilities and the universe of SIC and NAICS codes in the 2011 DMR and TRI data to the data in 

2009. ERG will also compare the number of facilities reporting, increases and/or decreases in 

loads, and pollutants reported for the top 95th percentile. 

 DMR Pollutant Loading Tool results. ERG’s quality review of the DMR Pollutant Loading Tool 

results includes the following checks. Each of these checks may result in a list of suspected 

inaccuracies. At the EPA WAMs direction, ERG will contact the impacted facilities and/or 

permitting authority for clarification. Based on these reviews, ERG will generate a list of 

identified inaccuracies and will review the list with the EPA WAM for inclusion or exclusion. At 

the EPA WAM’s direction, ERG will record the correction needed in a database corrections table 

that is included with the ARR as part of the 304(m) project record. ERG also delivers the entire 

list of database corrections to EPA’s Office of Compliance for inclusion in subsequent versions of 

the DMR Pollutant Loading Tool. 

- Accuracy of Facility Discharges. ERG will review and verify the accuracy of facilities’ 

discharges that had the greatest impact, in terms of TWPE, on total point source category 

loads and point source category rankings to identify possible calculation errors. ERG reviews 

the monitoring period and measurement data available in the tool for the top 95th percentile 

facilities to information from each facility’s NPDES permit and permit fact sheet to see if 

they match. For discharge amounts, ERG will compare those reported the DMR Pollutant 

Loading Tool to the permit limits for the pollutant to identify values that are an order of 

magnitude different since this is often an indication of a units error. 

- Accuracy of Point Source Category Discharges. ERG reviews for accuracy point source 

category discharges by verifying that pollutant discharges in the DMR Pollutant Loading 

Tool (for the top 95th percentile of facilities in the point source) were assigned to the 

appropriate point source category. ERG researches facility data using permit fact sheets, 

engineering textbooks and encyclopedias (such as Kirk-Othmer), and facility websites. ERG 

uses engineering judgment to determine if the pollutant discharge was reasonably associated 

with the point source category. 

- Reasonableness of Pollutant Loads. ERG reviews the output from the DMR Pollutant 

Loading Tool for those pollutant discharges with the highest TWPE (e.g., dioxins, 

polychlorinated biphenyls [PCBs], and mercury) to identify possible errors in recording units 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 34 of 109 

of measure. For these top pollutants, ERG identifies calculated discharges (at the facility and 

discharge pipe point) that are orders of magnitude higher than previous years’ discharges or 

discharges from other facilities within the same point source category. ERG then reviews 

quantities or concentrations and flows that the DMR Pollutant Loading Tool used to calculate 

the annual discharge. The DMR Pollutant Loading Tool includes a function to compare these 

measurements with measurements available in EPA’s Enforcement Compliance History 

Online (ECHO) web page. ERG also compares the data to permit documents. If the 

measurements are similar, ERG concludes that the output is acceptable. If the data do not 

match between the databases and ECHO, the data are noted as an inaccuracy to be reviewed 

with the EPA WAM. 

- Reasonableness of Facility Loads. ERG identifies facilities for review by whose pollutant 

discharges account for more than 95 percent of the TWPE for the point source category. ERG 

compares the facility’s DMR data to other available information, such as the facility’s 

NPDES permit and permit fact sheet and information from ECHO. If the data do not match 

between the database and ECHO, ERG will identify it as an inaccuracy to be reviewed with 

the EPA WAM. 

- Comparability. ERG compares the DMR data to prior years of data (such as the comparison 

of 2011 year data to 2009 year data) to identify pollutant discharges or wastewater flows that 

differed more than the year-to-year variation of other chemicals and facilities. ERG uses this 

comparison to determine if quantity, concentration, or flow corrections are needed for facility 

discharges with the highest TWPE. If the comparison is unavailable (e.g., the pollutant was 

not previously reported), ERG will contact the facility or permitting authority to solicit the 

needed data. 

 Review of Database Corrections from Past Years. ERG will review database corrections for 

past years (e.g., DMRLoads2009, DMRLoads2008, DMRLoads2007, PCSLoads2004, 

PCSLoads2002, and PCSLoads2000) to determine whether corrections made during previous 

reviews should apply to the current year DMR discharges. 

 Review of Current DMR Data. ERG will compare hand-calculated annual pollutant load 

results to loads calculated by the DMR Pollutant Loading Tool. 

 Review of Monitored Pollutant Discharges. ERG will review DMR pipe description 

information, ECHO discharge data, and the facility’s NPDES permit and permit fact sheet to 

identify monitored pollutant discharges that are: 

- Intermittent (e.g., occur after periodic operations, seasonal, or occur after a storm event); 

- Internal monitoring locations from which wastewater is combined with other waste 

streams and monitored again, resulting in double-counting loads; and 

- Not representative of point source category discharges (e.g., stormwater runoff from 

nonprocess areas, noncontact cooling water, or wastewater related to operations in another 

point source category). 
Revision No. 1 

Date: October 2013 

Section No. 3 

Page 35 of 109 

3.3.2 Other Existing Data Quality Review and Verification 

The criteria ERG will use to review and evaluate the quality of non-DMR existing data are summarized in 

Table 3-2, along with acceptance criteria based on how the data will be used. ERG will use other existing 

data on the 304 (m) project for the following uses: identifying industrial sector trends, describing 

wastewater characteristics, describing treatment system performance, estimating pollutant loads, 

estimating treatment costs, and describing human health and environmental impacts. Existing data sources 

will include discharge data (e.g., data received during stakeholder outreach and databases from other EPA 

offices, such as CDRR and TNSSS), articles and technical papers, industry/vendor data, and Internet 

information. Table 3-2 provides the specific circumstances that define when data meet acceptance criteria 

or when data are unusable. 

Data verification is the confirmation by examination and review of objective evidence that the data are 

complete, correct, consistent, and in compliance with technical requirements, established standards, and 

contractual requirements. To accomplish this, ERG will check the data source and/or data itself at the 

time of data collection against the quality criteria of Table 3-2. The ERG WAM will discuss any quality 

concerns from these reviews with the EPA WAM during their weekly status calls. ERG will also 

document in the project technical reports if any data or data sources do not meet the criteria. 

ERG may use data that do not meet the specifications if, for example, there are no other available data. 

ERG will discuss such data quality issues with the EPA WAM and obtain written approval before using 

these data in subsequent analyses. This approval will be added to the 304(m) project quality file. In the 

annual ARRs that support the annual preliminary or final effluent guidelines program plans, ERG will 

describe data quality, including any need to use other existing data that did not meet the quality 

acceptance specifications in Table 3-2. 

3.3.3 Achieving the Contract’s Overall Quality Objectives 

ERG’s procedures described in Section 3.1.1 result in identification of a variety of existing data of interest 

to EPA from numerous sources. ERG’s systematic review of those data against the data quality 

specifications presented in Table 3-2 help to ensure that the data are of a known and acceptable quality. 

Through these procedures, the existing data identified and collected for use will be unbiased and the 

resulting dataset used by EPA will meet EPA’s goal of objectivity. ERG’s procedures to ensure the 

security of existing data collected are described in Section 3.2. In addition, ERG’s procedures for the 

handling of existing data in spreadsheets and databases (presented in Section 7.0 and 8.0 below) helps to 

ensure data integrity and that data are not improperly altered during the collection and/or analysis process. 

The objective of collecting existing data that is usable is accomplished through ERG’s application of this 

section’s procedures, including methods for identifying usable sources of data and evaluation of its 

quality in consultation with the EPA WAM. 
Table 3-2. 304(m) Annual Review Data Sources Acceptance Criteria 

Type of Data Data Not Usable 

Data Usable for 

Industrial Sector 

Trends 

Data Usable to 

Characterize 

Wastewater 

Data Usable to 

Demonstrate 

Treatment System 

Performance/Efficiency

Data Usable to 

Estimate 

Pollutant Loads

Data U

Estimate 

Treat

Discharge Data (e.g., databases from other EPA offices such as WATERS) 

(1) Data collected by an unknown method or units undefined. (2) Data collected during upset conditions. (3) Represents a process that is not of interest (e.g., sanitary wastewater). 

Waste stream identified and analytes identified. 

Waste stream identified and analytes, units, and detection limits identified. 

(1) Represents full-scale system. (2) Paired influent and effluent data show that treatment system is well designed and operated. (3) Analytes identified and units included. 

Identified as discharge, analytes identified and units provided, approved analytical method used, represent current treatment operations. 

Dischargare not tyused for purpose. 

Articles / Technical Papers 

Information does not address Table 2-2 key questions 

Documented and reliable sources found through literature searches (e.g., peerreviewed journals and government publications). 

Waste stream identified and analytes, units, and detection limits identified. 

(1) Represents full-scale system. (2) Paired influent and effluent data show that treatment system is well designed and operated. (3) Data do not represent upset conditions. (4) Analytes identified and units and detection limits included. 

Identified as discharge, analytes identified and units provided, approved analytical method used, represents treatment operations. 

Cost comfully defidentified direct anindirect cannual oand maintenasystem scost year defined. 
Table 3-2. 304(m) Annual Review Data Sources Acceptance Criteria 

Type of Data Data Not Usable 

Data Usable for 

Industrial Sector 

Trends 

Data Usable to 

Characterize 

Wastewater 

Data Usable to 

Demonstrate 

Treatment System 

Performance/Efficiency

Data Usable to 

Estimate 

Pollutant Loads

Data U

Estimate 

Treat

Industry / Vendor Data 

Information does not address Table 2-2 key questions. 

These data are generally acceptable for relevant sources. 

Data not typically used for this purpose. 

(1) Represents full-scale system. (2) Paired influent and effluent data show that treatment system is well designed and operated. (3) Data do not represent upset conditions. (4) Analytes identified and units and detection limits included. 

Data not typically used for this purpose. 

Cost comfully defidentified direct anindirect cannual oand maintenasystem scost year identified

Internet Information (other than published articles or technical papers) 

Information does not address Table 2-2 key questions. 

These data are generally acceptable for relevant sources. 

Data not typically used for this purpose. 

Data not typically used for this purpose. 

Data not typically used for this purpose. 

Data not typically this purp
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 38 of 109 

Characterizing process wastestreams 

may include identifying the frequency 

and volume of generation, flowrates, 

pollutants in the wastestream, and 

pollutant concentrations. 

Characterizing wastewater treatment 

system performance may include 

identification of influent and effluent 

pollutant concentrations and/or loads 

and the percent removal of pollutants 

through the system. 

4.0 COLLECTION OF EXISTING DATA FOR USE IN EAD’S INDUSTRY STUDIES AND ELG

DEVELOPMENT

ERG collects existing data in support of all of EAD’s programs. Section 3.0 of this plan discusses existing 

data use for the 304(m) annual review process, while this section addresses existing data use in support of 

EAD’s studies and/or ELG development. Collection of existing data is necessary for the following 

reasons: 

 EAD programs require a variety of industry, process, treatment, and environmental-related data, 

not all of which is available through primary data collection activities; 

 Collection of existing data allows EPA to make more effective use of limited resources (in both 

time and budget) than collection of primary data alone; and 

 Existing data may provide qualitative and/or quantitative information that supports EAD’s 

primary data collection activities. 

This section details ERG’s procedures for (1) identifying and selecting existing data for use in support of 

EAD programs and (2) QA/QC that ERG follows to evaluate and document the quality of existing data 

collected and to describe the limitations of its use. 

4.1 Data Gathering

ERG and EPA use data collected from existing sources to support EAD’s industry studies and/or 

development of ELGs. These data are documented in study reports and public records for rulemaking 

activities and are subject to public review and comment. This section discusses ERG’s procedures for 

gathering existing data in support of EPA’s industry studies and ELG development programs. 

4.1.1 Data Needs and Potential Data Sources 

ERG begins gathering existing data first by working with EPA to define data needs for the work 

assignment/project and the intended use of the data. Data needs are identified by the EPA work 

assignment, ERG’s description of our technical approach in the work plan, through meetings and 

discussions, and through planning memoranda. As directed by the EPA WAM, ERG will prepare data 

collection memoranda to document the task specific approach for existing data collection. Under Contract 

EP-C-12-021, ERG will collect existing data to: 

 Create, revise, or support an industry profile; 

 Identify and characterize process wastestreams; 

 Identify applicable treatment technologies and/or 

pollution prevention practices for specific waste 

streams and/or pollutants of concern; 

 Characterize wastewater treatment system 

performance/efficiency; 

 Estimate baseline and option pollutant discharges; 

 Estimate treatment costs; 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 39 of 109 

 Identify environmental impacts associated with pollutant discharges; and 

 Estimate water column, sediment, and fish tissue pollutant concentrations. 

EPA can then use the data to identify and evaluate alternative regulatory approaches. Table 4-1 presents 

the data elements that ERG will collect for the listed existing data purposes and potential sources for 

them. 

Table 4-1. Existing Data Purposes and Data Elements Gathered 

Existing Data Purpose Data Elements Gathered Potential Data Sources 

Create, Revise, or Support An 

Industry Profile 

 Facilities in the universe of a 

specified point source category. 

 Facility location information. 

 Facility ownership information. 

 Facility size. 

 Facility operation descriptions (may 

be based on type of fuel use, type of 

product, type of operations). 

 Facility financial information 

(publicly available). 

 Existing agency databases. 

 Other governmental agency 

databases/lists. 

 Trade association lists. 

 Dun and Bradstreet. 

 Company/facility publications and 

websites. 

 Technical literature. 

 Company financial statements. 

Identify and Characterize Process 

Wastestreams 

 Industry descriptions. 

 Facility-level operation descriptions. 

 Process wastestream pollutants and 

concentrations. 

 Existing agency documents (e.g., 

sector notebooks, background 

documents, study reports). 

 Agency databases (e.g., DMR and 

TRI datasets). 

 Other governmental agency 

documents. 

 Technical literature. 

 Existing industry/facility site visit 

and sampling data. 

Identify Applicable Treatment 

Technologies and/or Pollution 

Prevention Practices for Specific 

Waste Streams and/or Pollutants 

of Concern 

 Industrial treatment technology 

descriptions. 

 Performance data on specific 

technologies or pollution prevention 

practices. 

 Pollution prevention practice 

descriptions by industry/operation. 

 Existing Agency documents and 

databases (e.g., study reports, 

NRMRL treatment performance 

database). 

 Other governmental agency 

documents and databases (e.g., TRI 

database, ATSDR studies). 

 Technical literature. 

 Conference papers. 

 Trade association publications. 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 40 of 109 

Table 4-1. Existing Data Purposes and Data Elements Gathered 

Existing Data Purpose Data Elements Gathered Potential Data Sources 

Characterize Wastewater 

Treatment System 

Performance/Efficiency 

 Treatment performance data 

including influent and effluent 

concentrations. 

 Percent reduction through treatment 

for a specified pollutant. 

 Analytical methods used. 

 Sampling and analytical QA/QC 

performed. 

 Existing Agency sampling data. 

 Existing Agency treatability studies. 

 NRMRL’s Treatability database. 

 Existing industry/facility sampling 

data. 

 Technical literature. 

 Vendor discussions/data/studies. 

Estimate Baseline and Option 

Pollutant Discharges 

 Industry effluent concentrations and 

flows. 

 Facility-specific effluent 

concentrations and flows. 

 Existing Agency sampling data on 

facility/industry effluent 

concentrations and flows. 

 Agency databases (e.g., DMR and 

TRI datasets). 

 Existing industry/facility sampling 

data on effluent concentrations and 

flows. 

 Technical literature. 

 Vendor discussions/data/studies. 

Estimate Treatment Costs Capital and/or annual costs for a 

specific technology, technology train, 

or pollution prevention practice. 

 Vendor cost data. 

 U.S. Army Corp of Engineers 

CAPDET model. 

 RS Means Construction Cost data. 

 Technical literature. 

Environmental Assessment 

Analysis 

 Pollutant toxicity. 

 Human health effects data. 

 Aquatic health effects data. 

 Linkage of facility to its discharge 

water body. 

 Receiving stream characteristics. 

 Documentation of environmental 

incidents/harm. 

 Agency studies and reports. 

 Agency databases (e.g., WATERs, 

303(d) list of impaired waters). 

 USGS databases. 

 National Hydrography Dataset. 

 Existing agency toxicity databases. 

 Other agency studies and databases. 

 Technical literature. 

 GIS coordinates for facilities and 

water bodies. 

 Conference proceedings. 

 Gray sources (such as newspaper 

articles or press releases). 

ERG identifies potential data sources through: 

 Identification by the EPA WAM and/or other Agency staff; 

 Identification by industry or other agency contacts; 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 41 of 109 

 Review of references cited in previously identified published literature; and 

 Literature search. 

Section 4.1.2 below presents details on ERG’s procedures for conducting literature searches and the 

search engines that we use to identify existing data sources. As directed by the EPA WAMs, ERG will 

document literature search methodology and results in memoranda or literature search review reports that 

specify data needs, key words used, sources identified, and ERG’s review of the source quality. 

4.1.2 Literature Searches 

ERG literature searches use a keyword list along with a search engine to identify technical papers/reports, 

studies, and peer-reviewed journal articles that document and/or provide data on a specific technical topic. 

In addition, as time is available, ERG will search for information from newspapers, environmental 

groups, industry organizations, and other non-peer reviewed information sources. ERG acknowledges that 

EPA considers these sources “gray” literature that may not be acceptable forms of formal documentation. 

However, ERG recognizes that these literature sources can provide useful information for identifying 

incidental and qualitative data and examples that support data from peer-reviewed information sources. 

For example, in conducting literature searches for environmental assessment analyses, ERG will often 

find environmental events reported in gray literature sources before an environmental impact is well 

documented in peer-reviewed journals or government reports. ERG intends to use gray literature to help 

highlight areas of interest and facilitate additional searches of peer-reviewed journals for more detailed 

information on the topic of interest. EPA may also direct ERG to use gray literature if other preferred 

sources of data are not available. 

ERG uses several different search engines to broaden the range of reference materials represented in the 

results. These search engines include: 

 Dialog (www.dialog.com, a licensed search product) – Dialog is a comprehensive search engine 

for searching articles and reports from thousands of in-depth repositories of scientific and 

technical data, patents, trademarks and other intellectual property data, real-time news feeds, 

newspapers, broadcast transcripts and trade publications, plus market research reports and analyst 

notes providing support for financial decision-making. 

 Scirus (www.scirus.com) – Scirus is a comprehensive science-specific search engine. Scirus 

provides access to a large database of scientific, technical, and medical journals. 

 Ingenta (www.ingentaconnect.com) – Ingenta is a scholarly research database that provides 

access to a large database of academic and professional research articles. 

 Google Scholar (http://scholar.google.com/) – Google Scholar is a general search engine of 

scholarly literature, including peer-reviewed papers, theses, books, abstracts and articles from 

academic publishers, professional societies, preprint repositories, and universities and other 

scholarly organizations. 

 Google (www.google.com) – Google is a general search engine of information readily available 

on the Internet. 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 42 of 109 

ERG will use a key word list developed for the project and will often pair multiple keywords to focus 

search results. ERG develops multiple keyword combinations using our best professional judgment and 

knowledge of the topic under investigation. ERG will document key words in a project planning 

document and/or in the literature review documentation as described in Section 4.1.5, unless directed by 

EPA not to do so. 

4.1.3 Criteria for Selecting Data Sources and Resolving Data Gaps 

Criteria that EPA and ERG use to evaluate candidate data sources include: 

 Relevance of the source to the data need; 

 Reliability of the source; and 

 Content of the source. 

Relevance. Data sources specific to the topic being investigated or containing data specific to the industry, 

operation, wastestream, pollutant of interest, and/or receiving water will be considered for use. Sources 

that most closely represent the topic/data of interest are the most relevant. 

Reliability. ERG and EPA consider the following factors when evaluating reliability of a data source: (1) 

sources that have been generated by EPA and/or other governmental agencies and subject to data review 

and assessment are considered to be the most reliable, (2) sources generated by entities with established 

knowledge in the topic area (e.g., studies conducted by industry experts, primary data generated by an 

industrial facility using documented and approved methods) are also considered by EPA to be reliable, 

and (3) sources that use unknown collection and data review procedures are considered by EPA as less 

reliable. 

Content. EPA also considers the source’s content in selecting it for use. Examples of source content can 

include extent of data (e.g., how many facilities does it cover, over what geographical area, other what 

period of time), age of the data, and level of documentation describing the generation of the data. 

EPA and ERG will meet to discuss the identified existing data sources for each project. Through 

discussion, EPA and ERG will decide whether sufficient relevant and reliable data sources exist and 

whether those sources contain sufficient data for the intended use. The gathering of existing data in 

support of EAD studies and ELGs is an iterative process and will often involve the following steps: 

 Identification of existing data sources; 

Assessment of their relevance, reliability, and content; 

Determination of whether data gaps exist; and 

Resolutions for filling data gaps. 

To resolve data gaps, EPA may instruct ERG to identify additional existing data sources, collect primary 

data, and/or develop assumptions or a methodology to address the data gap. 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 43 of 109 

4.1.4 Data Value Selection Approach 

As part of the data source selection and data gap evaluation described in Section 4.1.3, ERG will work 

with EPA to conduct data value selection, i.e., the selection of data elements of interest from the potential 

sources identified. Once ERG and EPA have screened the potential sources of needed data, we may find 

that: 

 More than one source offers the data element(s) of interest; 

 Only one source provides the data element of interest; or 

 No sources yield the data elements of interest. 

When more than one source offers the data element(s) of interest, the following data selection approaches 

are used: 

 If only one of the sources is deemed relevant and reliable, then that source will be used; 

 If more than one of the sources is deemed relevant and reliable, ERG will examine the range of 

values and will recommend an approach. EPA can then choose to average the data values from 

the multiple acceptable sources or can select a preferred source (where preference is given to 

values that best fit the data need); and 

 If none of the sources are deemed relevant and reliable, EPA may direct ERG to conduct further 

literature searches or to develop an assumption or methodology to estimate the needed value. 

When values are used from multiple existing data sources and when directed to do so by EPA, ERG will 

investigate the range of data values found, whether the data represent normal distributions, or whether the 

data appear to include non-normal distribution of values. In cases where multiple data points are normally 

distributed, averaging the data for an arithmetic mean is appropriate. In cases where multiple data points 

are non-normal in their distribution, then using a median of the data may be more appropriate. Finally, in 

cases where a single value is not required for the analysis, presentation of the value range may be most 

appropriate. ERG will work with the EPA WAM in deciding which approach to use for multiple 

acceptable values. 

When only one source offers the data element(s) of interest, the following data selection approaches are 

used: 

 If the source is deemed relevant and reliable, that source will be used; and 

 If the source is not deemed relevant and reliable, EPA may direct ERG to conduct further 

literature searches or develop an assumption or methodology to develop the needed data. 

When none of the potential data sources yield the data elements of interest, EPA may direct ERG to: 

 Conduct additional literature searches for a relevant and reliable source of the data element(s); 

 Collect needed data element(s) directly through primary data collection procedures (e.g., through 

a phone contact or sampling); or 

 Develop an assumption or methodology to develop the needed data. 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 44 of 109 

4.1.5 Data Gathering Documentation and Records 

The EPA WAM will determine the manner and format for literature review documentation. ERG will 

document the methodology used for identifying existing data sources, the criteria used in selecting data 

sources for further use, and the methodology for data selection in planning memoranda, data search and 

result reports, and final deliverables unless directed by EPA not to do so. 

ERG will obtain data from sources such as databases, publications, other EPA offices, government 

publications, and scientific literature. 

ERG staff will use the following procedures when collecting published data: 

 Collect full bibliographic information on all collected publications, including “date accessed” for 

information collected from the Internet. 

 Save web pages as PDF files, so that the data source can be added to the electronic project file. 

ERG staff will use the following documentation procedures when collecting data via telephone and email 

contact. 

 Document all telephone contacts using a Telephone Contact Report depicted in Attachment C. 

 Save all email contacts as PDF files, so that they can be added to the electronic project file. 

 Ensure that telephone contact reports and email records are added to the electronic project file. 

Data gathered will be documented by ERG project staff and saved as files in the project folder maintained 

electronically on ERG’s project server. ERG’s WAM in consultation with the EPA WAM will determine 

which data materials gathered during the project need to be included in any record that supports the study 

and/or ELG. 

4.2 Data Management

Specific projects may have a project file management plan that presents detailed procedures ERG will 

use for managing the project files assembled in support of the project and for submitting appropriate 

portions of these files to a docket (if applicable). For example, all engineering information collected, 

generated, and analyzed on the steam electric ELG project is handled and stored in a centralized record, 

as described in the Steam Electric Power Generating Project File Management Plan, dated September 25, 

2009. This centralized record contains all data collected or accessed from other sources and documents 

analyses of data collected (e.g., site visit reports, memoranda, industry profiles). The contents of the 

record are indexed using a MS Access® database. 

For projects without a specific file management plan, the ERG Program Manager and ERG WAM are 

responsible for ensuring project documents and records conform to technical requirements and quality 

system requirements specified in the work assignment or technical direction by the EPA WAM. Work 

assignment planning documents (e.g., the work plan and this PQAPP) identify the level of review 

required for project deliverables and who will conduct the review. This review is conducted using the 

quality assessment tools described in this PQAPP. The ERG Program Manager and ERG WAM are also 

responsible for ensuring that records and documents accurately reflect the completed work. 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 45 of 109 

For all EAD work assignments that ERG conducts under Contract EP-C-12-021, the ERG WAM assigns 

a Project Assistant the responsibility of maintaining the project file index and filing documents in a 

centralized project file (including the electronic project file). For non-EAD projects, project files may not 

be needed; if a non-EAD project requires a project file, the ERG WAM will assign project staff to create 

and maintain one. Stored records (hardcopy) are protected from damage, loss, and deterioration at all 

ERG locations by storing them on site in locked, climate-controlled file rooms. All ERG offices are 

equipped with smoke alarms and sprinkler systems in case of fire. Access to the record storage areas is 

limited to authorized personnel. Electronic project data, calculations, programs, models, databases, and 

written products are stored in identified project areas on ERG’s network. ERG conducts daily tape 

backups to protect against loss of electronic data. If a network server is damages or destroyed, electronic 

files will be recovered from back-up tapes and a new network server will be established. ERG makes 

back-up tapes of all server files on a weekly basis and stores them as follows: full back-up from the end of 

the previous week is stored in each office in a fire- and water-proof safe; full back-up from two weeks 

prior is stored in an off-site location. The ERG WAM will ensure that project team members are 

instructed on use of that project’s file management system. 

ERG procedures for the entry and/or uploading of electronic data from existing sources into spreadsheets 

and databases are further discussed in Sections 7.0 and 8.0. 

For Contract EP-C-12-021, ERG also stores inactive paper records in on-site storage rooms. Inactive 

electronic files are archived to tape or compact disk for long-term storage. ERG maintains the project files 

for EPA’s Office of Water projects until the materials can be: 1) boxed and sent to the Federal Records 

Center (FRC), 2) returned to EPA’s offices as directed, or 3) destroyed if directed to do so by EPA. 

Documents obtained from clients or other entities, or created by ERG, that fall under CWA CBI 

requirements are destroyed or returned to the appropriate person(s) in accordance with the entities’ 

written instructions. Other classified or confidential records are destroyed by shredding or incineration. 

4.3 Existing Data Quality Assessment

After identification and selection of existing data sources, EPA and ERG then review the data in those 

sources to assess its usability. The subsections below present and discuss quality criteria for existing data 

and how ERG will assess whether it is suitable for use. 

4.3.1 Quality Review of Existing Data Used in Characterizing Industry Point Sources, 

Wastestreams, and Treatment Technologies 

The criteria that ERG will use to evaluate the quality of literature sources used for characterizing industry 

point sources, wastestreams, and/or treatment technologies are: (1) accuracy, (2) reliability, and (3) 

representativeness. 

 Accuracy. ERG assumes that the underlying data and information contained in state and federal 

reports and peer-reviewed journal articles are accurate. ERG assumes that data from these sources 

are of sufficient quality to support industry and/or wastestream characterization and the 

performance of specified treatment technologies. ERG will present data that are outliers (e.g., 

data that are not consistent with the body of information collected from the literature review) to 

EPA for further direction on how to incorporate or exclude the data in the analysis. 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 46 of 109 

 Reliability. During the course of a literature search, ERG may encounter sources that contain 

limited documentation of the source of data/information (e.g., non-peer reviewed industry 

publication). These data sources represent a lower level of reliability than peer-reviewed 

literature, but may provide useful information for capturing the full range of processes and/or 

wastes generated by a specified industry point source or a range of information on specified 

treatment technologies. If ERG identifies limitations in the quality of information taken from 

other existing data sources (e.g., trade publications, vendor literature), ERG will communicate 

these data quality issues to EPA. ERG will also evaluate existing data for reliability based on the 

following factors: 

- The literature is clearly written, so that all assumptions and methodologies can be identified; 

- The variability and uncertainty (quantitative and qualitative) in the information or in the 

procedures, measures, or results are evaluated and characterized; and 

- The assumptions and methodologies are consistently applied throughout the analysis as 

reported in the source. 

 Representativeness. ERG will also evaluate the representativeness of existing data to ensure that 

it comes from the technology or process of interest. For example, pollutant identities must 

represent the process or wastestream from which they were discharged. ERG may flag a data set 

as questionable and subject it to further investigation if a detected chemical or its concentration is 

not expected in a waste stream or if an expected chemical is missing. ERG will communicate to 

the EPA WAM whether reported or calculated pollutant loads, plant wastewater flows, or other 

existing data reflect the range of expected values for specific operations/waste streams in the 

industry of concern. In addition, if the data provided by a data source are repeatedly found to be 

unrepresentative, ERG will discontinue use of the source and communicate these quality issues to 

EPA.. 

When existing data are collected, ERG will use the criteria in Table 4-2 to recommend the types of 

analyses existing data can be used for, if any. 
Table 4-2. Existing Data Acceptance Criteria for Industry Studies and ELG Development 

Type of Data Data Not Usable 

Data Usable 

for Profile 

Data Usable to 

Characterize InProcess Waste 

Data Usable to 

Demonstrate System 

Performance/Efficiency 

Data Usable t

Estimate Basel

Discharges 

Monitoring Data (e.g., NPDES permit, DMRs) 

(1) The plant has since changed operations (e.g., installed a new treatment system) since the data were collected. (2) Data collected during upset conditions. 

Waste stream identified and analytes identified. 

Waste stream identified and analytes, units, and detection limits identified. 

(1) Represent full-scale system operated at applicable steam electric power plant. (2) Influent and effluent data or percent removal identified. (3) Analytes identified and units included. 

Identified as discharge, analyteidentified and uniprovided, approveanalytical method used, represent current treatment operations. 

Other Monitoring Data (e.g., 308 Letter, Questionnaire, EPRI, In-Plant) 

(1) Data collected by an unknown method or units undefined. (2) Data collected during upset conditions. (3) Represents a process that is not of interest (e.g., sanitary wastewater). 

Waste stream identified and analytes identified. 

Waste stream identified and analytes, units, and detection limits identified. 

(1) Represents full-scale system operated at applicable steam electric power plant. (2) Influent and effluent data show that treatment system is well designed and operated. (3) Analytes identified and units included. 

Identified as discharge, analyteidentified and uniprovided, approveanalytical method used, represent current treatment operations. 
Table 4-2. Existing Data Acceptance Criteria for Industry Studies and ELG Development 

Type of Data Data Not Usable 

Data Usable 

for Profile 

Data Usable to 

Characterize InProcess Waste 

Data Usable to 

Demonstrate System 

Performance/Efficiency 

Data Usable t

Estimate Basel

Discharges 

Articles/Technical Papers 

Information does not address Table 2-2 key questions 

Documented and reputable sources found through literature searches (e.g., peerreviewed journals and government publications). 

Waste stream identified and analytes, units, and detection limits identified. 

(1) Represents full-scale system operated at applicable steam electric power plant. (2) Influent and effluent data show that treatment system is well designed and operated. (3) Data do not represent upset conditions. (4) Analytes identified and units and detection limits included. 

Identified as discharge, analyteidentified and uniprovided, approveanalytical method used, represent treatment operatio

Industry / Vendor Telephone Queries 

Information does not address Table 2-2 key questions. 

These data are generally acceptable for relevant sources. 

Data not typically used for this purpose. 

(1) Represents full-scale system operated at applicable steam electric power plant. (2) Influent and effluent data shows that treatment system is well-designed and operated. (3) Data do not represent upset conditions. (4) Analytes identified and units and detection limits included. 

Data not typically used for this purp

Internet Information (not published articles or technical papers) 

Information does not address Table 2-2key questions. 

These data are generally acceptable for relevant sources. 

Data not typically used for this purpose. 

Data not typically used for this purpose. 

Data not typically used for this purp
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 49 of 109 

Except as noted in Section 3.0, ERG will not review the quality of existing data contained in EPA 

databases (e.g., PCS, TRI, ICIS-NPDES) or other databases provided by federal or state agencies but will 

describe known limitations of the data in reports on which they are based. For existing data contained in 

databases provided by trade organizations (e.g., America’s Natural Gas Alliance), ERG will evaluate the 

source of the data and discuss with the EPA WAM the suitability of the data for the intended purpose. If 

ERG identifies limitations in the quality of the data from published literature or federal/state generated 

databases that may impact decisions regarding the overall direction of further data collection efforts or the 

quality of deliverables, ERG will communicate these issues to the EPA WAM. Data limitation 

discussions will identify where data gaps exist (e.g., in PCS, major facilities are required to submit data, 

but PCS data may not represent all facilities in a given geographic region because minor dischargers often 

are not included) and a description of data reporting requirements (e.g., TRI does not require facilities to 

sample and analyze waste streams to determine the quantities of toxic chemicals released, so the data 

included in TRI may be estimates of releases based on mass balance calculations, published emission 

factors, site-specific emission factors, or other approaches). 

4.3.2 Quality Review for Existing Data Used in Environmental Assessment Analysis 

In addition to the analyses listed in Table 4-2, ERG will use existing data to support qualitative and 

quantitative assessments of the impact of industry discharges on the environment (e.g., environmental 

modeling to predict receiving water concentrations and fish and wildlife tissue concentrations). 

4.3.2.1 Qualitative Analyses 

Qualitative analyses provide a descriptive discussion of environmental impacts without quantifying the 

specific impacts. Qualitative impact analyses will rely primarily on existing data collected as part of a 

literature review. The criteria that ERG will use to evaluate the quality of literature sources used for 

qualitative analyses are: (1) accuracy, (2) reliability, and (3) representativeness. 

 Accuracy. ERG assumes that the underlying data and information contained in state and federal 

reports and peer-reviewed journal articles are accurate. ERG assumes that data from these sources 

are of sufficient quality to identify potential environmental impacts associated with industry 

discharges. ERG will present data that are outliers (e.g., data that are not consistent with the body 

of information collected from the literature review) to EPA for further direction on how to 

incorporate or exclude the data in the analysis. 

 Reliability. During the course of the literature search, ERG may encounter sources that contain 

limited documentation of the environmental impact claim (e.g., unidentified speaker at a 

stakeholder meeting or a local resident commenting in a newspaper article). These data sources 

represent a lower level of reliability than peer-reviewed literature, but may provide useful 

information for capturing the full range of environmental impacts associated with industry 

discharges. If ERG identifies limitations in the quality of information taken from other existing 

data sources (e.g., websites, newspapers, and stakeholder meetings), ERG will communicate 

these data quality issues to EPA. ERG will also evaluate existing data for use in qualitative 

analyses for reliability based on the following factors: 

- The scientific work is clearly written, so that all assumptions and methodologies can be 

identified; 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 50 of 109 

- The variability and uncertainty (quantitative and qualitative) in the information or in the 

procedures, measures, methods, or models are evaluated and characterized; and 

- The assumptions and methodologies are consistently applied throughout the analysis as 

reported in the source. 

 Representativeness. ERG will evaluate existing data for use in qualitative analyses for 

representativeness based on whether they represent: 

- The types of environmental impacts caused by wastewater discharges from the industry of 

interest; 

- The range of concentrations observed in industry discharges; and 

- The environments affected by industry discharges. (e.g., rivers, lakes, and wetlands). 

Information on environmental impacts in all environments may not be readily available in the published 

literature. When a subpopulation is selected for analysis (e.g., lakes and reservoirs), ERG will clearly 

describe the subpopulation and resulting limitations of the analysis to EPA. 

4.3.2.2 Quantitative Assessments 

ERG may also use existing data to support quantitative assessments of the environmental impacts of 

industry’s discharges through the use of environmental models. Input values for the environmental 

models may include: 

 Pollutant loadings from waste streams of interest; 

 Discharge flow from discharging facilities; 

 Average annual receiving water stream flow; 

 Partition coefficients for pollutants of interest; 

 Bioaccumulation factors for pollutants of interest for different trophic levels of fish; 

 Average weight for wildlife species of interest; 

 Average water ingestion rate for wildlife species of interest; and 

 Food ingestion rate of prey in each fish trophic level for wildlife species of interest. 

ERG will obtain input values and parameterize the environmental models based on data from existing 

data sources, including: 

 Peer-reviewed literature; 

 Government documents; 

 Academic reports; 

 Publicly available GIS datasets from government organizations (e.g., NHD Plus); and 

 Databases provided by federal or state agencies (e.g., USGS stream monitoring data). 

ERG will not review the quality of existing data contained in government GIS systems or databases, but 

will describe known limitations of the data in reports on which they are based. If ERG identifies 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 51 of 109 

limitations in the quality of the data from published literature or federal/state generated GIS or databases 

that may impact model predictions, ERG will communicate these issues to the EPA WAM. In addition to 

the quality criteria described for the qualitative environmental assessment analyses, ERG will also use the 

following acceptance criteria to evaluate use of existing data for environmental modeling analyses: 

 The data are from a peer-reviewed, government, or academic source; 

 The geographical area of which the data are collected are representative of the goal of the 

modeling analysis; 

 The exposure scenarios and concentrations are either directly from relevant facilities or are 

representative of the types of waste streams under consideration for the work assignment or 

project; 

 The data sets are representative and reflect the ecological or human population of interest; 

 The scientific and technical procedures, measures, methods, or models employed to generate the 

information are well documented and reasonable for, and consistent with, the intended application 

within the environmental model; and 

 The data are comparable to other sources. 

4.3.3 Data Quality Verification and Reporting 

Data verification is the confirmation by examination and review of objective evidence that the data are 

complete, correct, consistent, and in compliance with technical requirements, established standards, and 

contractual requirements. To accomplish this, ERG will determine at the time of collection if collected 

data sources and data elements meet the data quality specifications set out in sections 4.1.3, 4.3.1, and 0. 

ERG will compare data sources and data elements to the quality specifications and document the results 

as part of the project file through notes in spreadsheets and/or databases, as part of project notes, 

memoranda, and/or documents, or when directed in specific data quality review reports. ERG will also 

discuss with the EPA WAM if any data or data sources do not meet the specifications. A review of data 

usability is also conducted during data analysis and evaluation as part of the interactive process between 

ERG and EPA. 

ERG may be directed by EPA to use data that do not meet the specifications if, for example, there are no 

other available data. ERG will discuss such data quality issues with the EPA WAM and obtain written 

approval before using these data in subsequent analyses. This approval will be added to the project quality 

file. In its final project documentation, ERG will describe data quality reviews, including any need to use 

data that did not meet quality acceptance specifications. 

As discussed in Section 2.8, ERG will also provide the ERG WAM with a monthly report of the status of 

QA activities. 

4.3.4 Achieving the Contract’s Overall Quality Objectives 

ERG’s procedures described in Section 4.1result in identification of a variety of existing data of interest 

to EPA from numerous topic-specific sources. ERG’s systematic review of those data against the quality 

objectives described in Sections 4.3.1 and 0 helps to ensure that the data are of a known and acceptable 

quality. Through these procedures, the existing data identified and collected for use will be unbiased and 
Revision No. 1 

Date: October 2013 

Section No. 4 

Page 52 of 109 

the resulting dataset used by EPA will meet EPA’s goal of objectivity. ERG’s procedures to ensure the 

security of existing data collected are described in Section 4.2. In addition, ERG’s procedures for the 

handling of existing data in spreadsheets and databases (presented in Sections 7.0 and 8.0 below) helps to 

ensure data integrity and that data are not improperly altered during the collection and/or analysis process. 

The objective of collecting existing data that is usable is accomplished through ERG’s application of this 

section’s procedures, including methods for identifying usable sources of data, procedures for its proper 

maintenance and storage, and evaluation of its quality in consultation with the project-specific EPA 

WAM. 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 53 of 109 

5.0 GENERATION OF PRIMARY DATA THROUGH QUESTIONNAIRES AND SURVEYS 

ERG will support EPA in developing, administering, and analyzing results of electronic and/or paper 

questionnaires that are sent to facilities within specified point source or industry categories as part of ELG 

development. In addition, EPA may also direct ERG to develop and administer smaller survey efforts to 

collect specific primary data from a subset of entities or may direct ERG to collect primary data through a 

site visit. Note: A survey of nine or fewer entities for the same information does not require an 

Information Collection Request (ICR). 

A point source or industry category subject to a questionnaire effort will be defined by the EPA WAM. 

ERG may assist EPA in defining the sampling frame based on publicly available data and data collected 

during any preliminary or detailed study. EPA’s statistics team is responsible for selecting sample 

populations, and ERG will use the statistics team’s input to develop the questionnaire distribution list. 

Based on past questionnaire efforts, ERG has developed mailout and helpline procedures to appropriately 

distribute questionnaires as well as data review procedures and guidelines to appropriately handle 

responses, review, and follow-up. ERG will compile responses into a database for use in technical 

analyses to answer the Table 2-2 key questions that are presented in Section 2.2. 

5.1 Data Gathering

This section describes data generation and acquisition via a questionnaire, survey, or site visit effort and 

covers the following four elements: 

 Data collection process design; 

 Data collection methods; 

 Questionnaire data handling and custody; and 

 Non-direct measurements. 

5.1.1 Questionnaire Data Collection Process Design 

The design of a questionnaire data collection effort will depend on the following factors: 

Type(s) of Questionnaires Needed for a Specific Industry Category. EPA may choose to distribute 

screener and/or detailed questionnaires. EPA uses screener questionnaires to quickly identify entities that 

are within scope of a regulatory effort and/or to identify a subset of entities that should participate in 

detailed surveys to help reduce the burden of a full survey effort. As an example, for the airport deicing 

point source category, EPA conducted a screener survey of airlines and a detailed survey of both airlines 

and airports. The results of the airline screener survey were used to identify a subset of airlines for the 

detailed questionnaire. 

Types of Data Needed for a Specific Industry Category. EPA generally develops multiple questionnaire 

parts and sections as needed so that questions on a similar topic can be grouped together. As an example, 

the Steam Electric questionnaire consisted of nine parts (Parts A through I) with the various parts 

targeting certain types of power plants or power plant operations. Also, in addition to the engineering and 

other technical data, the steam electric questionnaire included a section to obtain financial and economic 

information for use by EPA’s financial contractor. EPA distributed the questionnaire to all coal- and 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 54 of 109 

A questionnaire sample frame is the 

list of facilities within a population 

that can be sampled. 

petroleum coke-fired power plants and a statistically sampled subset of the rest of the steam electric 

industry, including oil-fired, gas-fired, and nuclear power plants. Questionnaires may also be directed at 

specific types of entities (e.g., POTW recipients to assess the impacts of pretreatment) or specific types of 

information (e.g., Steam Electric industry leachate monitoring questionnaire). 

Form of the Questionnaire(s). EPA has used both paper (hardcopy) and electronic questionnaires in past 

ELG development efforts. Because of the advantages of direct data importing with the use of an 

electronic questionnaire, ERG anticipates that future questionnaire efforts under Contract EP-C-12-021 

will be conducted using an electronic form. However, the discussion of data gathering elements included 

in this PQAPP applies to both types of questionnaire formats. In addition, for small directed data 

collection efforts, EPA may chose to conduct telephone surveys. In these cases, ERG will collect 

responses onto a data collection form developed specifically for the purpose. 

Mechanisms for Handling CBI Responses in the Questionnaires. Any technical and/or financial 

information designated CBI in a questionnaire response will be handled by ERG according to ERG’s CBI 

security plan entitled Security Plan for Handling Confidential Business Information Under the Clean 

Water Act Contract EP-C-12-021, dated November 9, 2012 and approved by Ahmar Siddiqui, EPA’s 

Document Control Officer, on January 9, 2013. EPA’s questionnaire data collection design needs to 

address how CBI will be identified and sent by the survey recipient to EPA. 

ERG will work with the EPA WAM, EPA statistics team, and any other relevant contractors to support 

the development of the questionnaire process design. EPA will lead the design development effort 

through a series of discussions/meetings and interim deliverables whereby each of the factors listed above 

are assessed for the specific industry category under review. ERG, EPA, and other contractors will 

document the resulting questionnaire collection design in a series of planning memoranda and/or 

documents. This documentation will include the following: 

 Description of the questionnaire sample frame 

development; 

 Summary of the systematic planning process for the 

questionnaire data collection; and 

 Information Collection Request (ICR) Supporting Statement. 

5.1.2 Questionnaire Data Collection Methods 

EPA’s questionnaires for industry studies or ELG development collect information for answering the key 

questions outlined in Table 2-2 of Section 2.2. ERG supports EPA in developing the Information 

Collection Request (ICR) supporting statement and respondent burden estimates which are approved 

through the Office of Management and Budget (OMB) prior to a data collection from more than nine 

entities. ERG will also support EPAin developing the draft and final versions of a questionnaire. EPA 

solicits comment on their draft questionnaires from other EPA offices, states, environmental groups, 

industry, and trade associations and the questions are made publicly available for comment. OMB also 

completes a thorough review of any EPA questionnaire that is subject to the ICR process. ERG supports 

EPA by revising the draft questionnaire, including specific questions, formatting, and response patterns, 

based on all comments received to make sure the questions capture the specific information that is 

required for the intended rulemaking while minimizing overall burden to the respondents. 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 55 of 109 

EPA’s information collection activities are conducted under authority of Section 308 of the Clean Water 

Act (Federal Water Pollution Control Act, 33 U.S.C. Section 1318). All facilities that receive a 

questionnaire must respond to it. ERG mails questionnaires to those facilities identified through the 

sample frame development process. ERG also works with existing data sources and any applicable trade 

associations to review the mailing list to ensure that facilities covered by the industry category are 

included and that appropriate respondent addresses/contacts are listed. 

All questionnaire responses must include a certification statement signed by a responsible corporate 

official or his/her authorized representative. EPA will ensure compliance with the information request by 

using a variety of means, including, as necessary, follow-up phone calls and letters and appropriate 

sanctions as provided by law. ERG will support EPA, as directed, with follow-up telephone calls and 

letters. For electronic questionnaire files (e.g., a questionnaire file in Excel®), the questionnaires may be 

copied onto electronic media by the Government Printing Office (GPO) or at ERG (within the constraints 

of the contract printing clause). In these cases, the electronic media will be clearly labeled with the title 

and version of the questionnaire. For hardcopy questionnaires, the questionnaire may be printed by GPO 

or at ERG (within the constraints of the contract printing clause). In these cases, the hardcopy forms will 

be clearly labeled with the title and version of the questionnaire. ERG then mails the questionnaires (on 

electronic media or paper copy) to the intended recipients. 

ERG follows specific procedures to ensure each facility receives the correct version of the questionnaire. 

ERG uses a mailout QA checklist (see Attachment E for an example of a mailout QA checklist) to 

document that the assembly of each mailout package is correct and complete. ERG develops the mailout 

QA checklist, customizing it as needed based on the number and types of questionnaire inserts for the 

mailout packages. ERG also documents which package is sent to each respondent. ERG will create and 

use an industry-specific Electric Tracking Database to document questionnaire delivery using a delivery 

service with tracking capabilities, such as Federal Express. This database is also intended to aid ERG in 

keeping track of questionnaire review, discussed in Section 5.3. ERG QC activities associated with the 

mailout procedures include: 

 Checking each printed mailing label against the finalized mailing list and “checking off” the 

appropriate column on the mailout QA checklist; 

 Checking 10 percent of the electronic media containing the questionnaire (or paper 

questionnaires) after receipt from EPA to make sure that all of the files required for the version 

are present and readable; 

 Assembling the mailout package, including the questionnaire cover letter and enclosures, the 

appropriate questionnaire version, and mailing label, and “checking off” on the mailout QA 

checklist that all respondent IDs match on all mailout documents; 

 Performing a 100 percent second check that the assembled package includes the correct cover 

letter, enclosures, questionnaire, and return label; 

 Tracking all questionnaire deliveries through Federal Express tracking mechanisms to confirm 

that the questionnaire was delivered and electronically transferring FedEx information into the 

industry Tracking Database; 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 56 of 109 

 Tracking all returned questionnaires to identify respondents that did not respond or that provided 

incomplete responses; and 

 Initial follow-up via phone and written communications with respondents who did not respond or 

respondents providing incomplete responses. 

ERG will coordinate with the EPA WAM on any further communication for non-responsive facilities. 

These communications may include a series of late letters or follow-up calls/letters from EPA requesting 

completion and return of the survey or any further follow-up necessary for questionnaire review. 

All questionnaire recipients will have the option to identify the information or part of the information 

submitted as CBI. A “CBI” checkbox is typically used next to each question on the survey so that the 

respondent has complete control over which questions will be handled as CBI. See Section 5.1.3, for 

handling procedures of completed questionnaires, including those marked CBI. 

5.1.3 Questionnaire Data Handling and Custody 

Completed questionnaires will either be mailed directly to ERG or transferred to ERG by EPA, where 

they will be logged into the industry tracking database and transferred (data entry or data upload) into an 

industry electronic questionnaire database. See Section 5.2 for a discussion of the questionnaire database. 

Original completed questionnaires will be logged into ERG’s Project File Index and stored as CBI in the 

electronic project file (for electronic responses) and in locked project file record rooms located in ERG’s 

offices (for hardcopy questionnaires, electronic media, hardcopy or attachments). All ERG offices are 

equipped with smoke alarms and sprinkler systems in case of fire. Access to the record storage areas is 

limited to authorized personnel and ERG institutes a signout procedure for the removal and return of any 

hardcopy or electronic media materials from the file record rooms. 

In order to ensure data quality, ERG will not change original responses of the questionnaire during review 

and follow-up. ERG will create working copies to complete these changes. See Section 5.3.1 for further 

discussion of questionnaire data receipt. Any paper copies received will be stored and used only as a 

reference to the original data submitted by the respondent. 

5.1.4 Non-Direct Measurements 

While preparing an industry survey effort, ERG will acquire, organize, and analyze existing data to 

characterize facilities for selection of a statistical subset of the population of interest that will receive the 

questionnaire, when applicable (e.g., some survey efforts cover the full population and will not require a 

statistical subset). EPA may direct ERG to compile this existing data into a sample frame database that 

includes all facilities potentially in scope for the questionnaire. Section 8.0 discusses ERG standard 

procedures for developing databases. Section 4.0 presents ERG’s data gathering, data management, and 

quality assurance procedures for existing data. 

5.1.5 Site Visit Primary Data Collection 

In addition to primary data collection through questionnaires and telephone surveys, ERG will also collect 

primary data through site visits. The types of data collected during site visits will support answering the 

questions in Table 2-2 and may include: 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 57 of 109 

 Facility specific information that characterizes the operations and wastewaters generated and 

discharged; 

 Wastewater management and treatment operations; 

 Potential locations for sampling points; and 

 Facility specific information on costs, loads, and discharge pollutants, flows, and concentrations. 

ERG staff will document information collected during site visits in a Site Visit Report (SVR). 

5.2 Questionnaire Data Management

The preferred format for questionnaires is an electronic information request tool (electronic 

questionnaire), which allows respondents to record their responses in a computer file and submit them to 

EPA on an electronic medium such as a CD, DVD, or flash drive. For future questionnaire activities 

conducted under Contract EP-C-12-021, ERG will work with EPA as part of the questionnaire planning 

and ICR process to assess the most appropriate form of the questionnaire (i.e., either an electronic form or 

a paper (hardcopy) form). 

Upon receipt of a returned questionnaire electronic media and/or paper survey, ERG will follow 

procedures outlined in the data review guidelines for handling the completed questionnaire, previously 

discussed in Section 5.1.2. ERG will conduct the required review of all responses, complete any 

necessary follow-up with facilities, and finalize any of the outstanding documentation regarding 

questionnaire review and follow-up. 

For primary data collected as part of a site visit, ERG will document the information received in a Site 

Visit Report (SVR), which is reviewed by both EPA and the facility visited. Data collected during a site 

visit may be used to inform the sampling and analysis process and may also be used along with other 

existing data in technical analyses. 

5.2.1 Transfer of Electronic Questionnaire Responses into a Questionnaire Database 

ERG will transfer all questionnaire responses (CBI and non-CBI) received into one questionnaire 

database. The questionnaire database as a whole will be handled as CBI until such later time as a 

sanitization plan is made to create a non-CBI version for the public docket. The database will be stored 

electronically on ERG’s CBI server. ERG will prepare a data element dictionary to document the database 

structure and to serve as a guide for database users. 

In order to transfer responses from MS Excel®-based electronic questionnaires into the questionnaire 

database, ERG will create a program that references mapped answer cells from the base questionnaire 

files and transfers them into specific locations/tables in the database. For open-ended questions, ERG will 

transfer the responses into a text field of the database. ERG will map every worksheet of the electronic 

questionnaire and thoroughly check all mapped references in each worksheet before distribution. 

ERG will perform several small “test runs” to pull different parts of an electronic questionnaire into the 

database and will continue test runs until all data are entered into the database without error. ERG will 

then pull in a complete “mock-up” questionnaire to check that all mapping references and data correctly 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 58 of 109 

transfer into the database to the correct table. ERG will complete a 100 percent check of the resultant test 

database to ensure correct data transfer. 

ERG will also complete a 100 percent check of the transfer of data from the first questionnaire response 

received for each version of the questionnaire. ERG will notify the EPA WAM of the results of the 

transfer. If mapping errors are detected (e.g., information is populated in the wrong fields or incorrect 

tables), ERG will document the error, correct it, and continue to perform 100 percent checks on additional 

questionnaire responses until all data are transferred correctly. If data transfer errors are detected (e.g., 

data type mismatch, incorrect file name), ERG will document the error, correct it, and re-extract the data 

from that file. 

ERG developed the following aspects of the data transfer program to efficiently extract data and minimize 

back end work in the database. 

Survey Review Table. The survey review table is a table in the database that includes a listing of all 

respondent IDs and the status of their completeness check and review. The table includes several 

fields that are utilized to document complete and successful data transfer. The first field, marked by 

the survey reviewer, flags the reviewed questionnaires that are ready for data extraction. After a 

successful data extraction, the program removes the flag that marks the questionnaire as ready and the 

date and time of transfer are recorded in another field. However, if the data transfer is not successful, 

the flag that marks the questionnaire as ready is not removed, and another flag is added indicating an 

error. 

Error Log. The error log is a table in the database that automatically updates during data transfer. If 

any transfer errors occur (i.e., data type mismatch, incorrect file name), the error log will populate the 

respondent ID, table name, and field name where the error has occurred. 

True Zero Values. Due to the nature of electronic questionnaire forms, all blank text and numeric 

fields are stored as “0”. ERG developed a process to distinguish between the blank data and true zero 

values entered by the respondent. Because both values are entered as zero into the database, ERG 

survey reviewers will change true zero responses to “-555” during technical review (i.e., before 

extraction of the data). ERG’s data extraction program will change any remaining blank or zero 

values to “-999.” After data extraction is complete, blank fields will show as “-999” and true zero 

fields will show as “-555.” The transfer program will then automatically update all “-555” values to 

zero. This process will represent the actual data entered into the questionnaire. 

Blank Tables. Certain aspects of an electronic questionnaire inevitably result in blank responses. 

These include skip patterns and tables where respondents answer only applicable rows. In order to 

avoid a database full of blank data, ERG’s extraction program will be tailored to each questionnaire 

to skip these rows of blank data. This will result in a database populated with only the information 

from facilities that have responded to a specific part, section, or subset of questions (depending on the 

skip patterns throughout the questionnaire). 

To ensure that the database accurately reflects responses, ERG will perform a 100 percent comparison of 

the information transferred into the database by section of the questionnaire to information recorded in the 

electronic survey file, for a pre-determined percent of responses. This comparison will also check that the 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 59 of 109 

survey review and error log tables accurately reflect the results in the database. This QA review will 

include checks that blank data are recorded as “-999”, true zero responses are recorded as zero in the 

database, and that blank rows/tables of information not required (e.g., due to skip patterns), are not 

transferred into the database. This verification step will confirm that the electronic transfer operations are 

working correctly. Other typical database QC checks are described in Section 8.0. 

Following technical review and follow-up of the questionnaire responses, ERG will transfer data into the 

questionnaire database. ERG does not expect to manually enter any data, as a result of the electronic data 

transfer. However, in the event any data do require manual data entry, ERG will perform a 20 percent 

check (or other percentage as specified by the EPA WAM) to ensure it has been accurately transcribed 

from hard copy. 

5.2.2 Transfer of Paper Questionnaire Responses into a Questionnaire Database 

ERG will transfer all paper (hardcopy) questionnaire responses (CBI and non-CBI) received into one 

questionnaire database. The questionnaire database as a whole will be handled as CBI until such later 

time as a sanitization plan is made to create a non-CBI version for the public docket. The database will be 

stored electronically on ERG’s CBI server. ERG will prepare a data element dictionary to document the 

database structure and to serve as a guide for database users. 

Following technical review and follow-up of the questionnaire responses, ERG will manually enter 

information from the paper working copy questionnaires into the database. ERG will generally use a 

double data entry process to identify any data entry errors. In the double data entry process, the responses 

from the same questionnaire are manually entered by two different staff and a computer program 

compares the entered data to ensure they match. The double data entry check will generate an error report 

that will document any issues that require resolution. An error resolution form will be created to 

document any fixes that are required for these databases. Database revisions from this process are also 

documented into a questionnaire database revision/correction spreadsheet. As with other documentation, 

the error report document and revision/correction spreadsheet will be submitted to EPA for approval prior 

to their use and the final documentation will be placed in the record as directed by the EPA WAM. 

5.2.3 Questionnaire Data Documentation 

Documentation for a questionnaire effort includes the questionnaire responses on electronic media or 

hardcopy, revisions or edits made by the response reviewer in the working copy of the questionnaire, the 

completed questionnaire review checklist, and documentation of communications with respondent 

contacts via a helpline or follow-up contact. These documents, except for the original electronic media or 

hardcopy response, will be stored in a working file specific to each respondent on ERG’s CBI server (or 

non-CBI server if no CBI data are provided). The original electronic media or hardcopy response will be 

stored in the ERG project archive. The questionnaire review task leader or his/her designee will inspect 

the questionnaire review documentation to ensure that all records are legible and complete. ERG will log 

these records into the project file and copies of these records will be provided to EPA as requested. 

ERG will use standardized practices for database development and database queries while executing the 

work described in this plan. Section 8.3.1 describes database development quality control checks and 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 60 of 109 

Section 8.3.2 describes database query quality control checks. These practices include requirements for 

identification and control of different versions of the database and its tables. These requirements include: 

 Assigning a unique name and number to the revised version of any database tables where 

revisions have taken place and maintaining a log of database table revisions. 

 Assigning a unique name and number to versions of the database. Also, maintaining a log that 

identifies database revisions and describes the changes made to the different versions. This log 

will be a table within the database. 

ERG will use data in the industry questionnaire database to perform specific analyses. ERG will construct 

database queries to perform these analyses, following the database query quality control procedures 

described in Section 8.3.2. 

5.2.4 Management of CBI Data in a Questionnaire 

Information collected through a questionnaire claimed as CBI by the survey respondent will be handled 

according to the procedures outlined in ERG’s CBI security plan entitled Security Plan for Handling 

Confidential Business Information Under the Clean Water Act Contract EP-C-12-021, dated November 

9, 2012, and approved by Ahmar Siddiqui, EPA’s Document Control Officer, on January 9, 2013. Only 

staff demonstrating their understanding of CBI handling procedures by passing a written test will be 

permitted to access CBI on this contract. 

5.3 Quality Assessment

This section describes ERG’s QC procedures for primary data collection through a questionnaire along 

with the data quality criteria that ERG will assess for acceptance and use of those data in an ELG 

development project. ERG will assist EPA in the review of all submitted questionnaires. As described in 

Section 5.3.1, ERG develops Data Review Guidelines to review questionnaires using several methods to 

improve completeness, consistency, and reasonableness of questionnaire responses. ERG typically 

conducts a question-by-question technical data review of all survey responses using the technical review 

checklist that defines the data acceptance criteria. ERG expects the project-specific technical review 

checklist to be updated periodically as the questionnaire review process evolves. Questionnaire data 

acceptance criteria are outlined in Section 5.3.2. ERG anticipates that the only circumstance in which a 

question response will be rejected would be a failure to resolve significant data issues with the respondent 

contact. 

5.3.1 Questionnaire Quality Control 

Helpline. As directed by the EPA WAM, ERG will operate a questionnaire helpline to offer assistance to 

entities responding to a questionnaire. Dependent on EPA’s direction, questionnaire respondents will be 

able to call in or email specific questions regarding the questionnaire and receive guidance on how to 

respond. Helpline contact information will be provided with questionnaire instructions. ERG will operate 

the helpline to ensure quality and completeness of questionnaire responses. ERG will develop helpline 

guidelines (which are tailored to a specific survey) so that information provided via the helpline is correct 

and consistent. The helpline guidelines also provide instructions for correctly documenting any helpline 

communication with a respondent. ERG will develop survey-specific helpline guidelines that include: 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 61 of 109 

Completeness Check 

 Check that the respondent has provided 

all requested contact information and 

respondent identification (e.g., facility 

name and respondent ID on each 

worksheet of the questionnaire); 

 Check that the respondent returned all 

applicable parts of the questionnaire; 

 Check that response data submitted 

includes all necessary files and that all 

parts of the file are present and able to be 

viewed/accessed; and 

 Check that all applicable questions are 

completed on each worksheet/section of 

the questionnaire. 

 Check that the certification signature is 

returned and complete. 

 Requirements for determining if a respondent needs to complete certain sections of the 

questionnaire; 

 Instructions/clarification on how to complete any part, section, question, or table; 

 Checklists for all required information for block/water balance diagrams (as appropriate); 

 Acceptable units of measure for specific questions; and 

 A log of responses to all issues encountered by respondents during questionnaire completion. 

Both a telephone line and email helpline are provided for respondents to use to receive help in completing 

the questionnaire. The helpline log contains two logs, one for the telephone helpline and one for the email 

helpline. Any phone or email communications with facilities will be documented in the helpline log. Each 

phone or email message will be represented by one line in the helpline log. Each message will be 

identified by respondent ID, date, time, and assisting helpline personnel. This helpline log will assist ERG 

in quickly searching past questions and responses while operating the helpline. As directed by the EPA 

WAM, the helpline log will also serve as a template to develop a frequently asked questions (FAQ) 

document that will be placed on EPA’s website and periodically updated to assist respondents while 

completing the questionnaire. The final FAQ document will ultimately be submitted to the public docket 

for the industry rulemaking. 

Data Receipt. ERG has developed a procedure to ensure the proper handling and storage of questionnaire 

responses. This procedure will help ensure that all questionnaire responses are received, tracked, saved, 

and copied to ERG’s CBI network. The process ERG will follow upon initial receipt of any 

questionnaires administered under this contract includes: 

 Update the Survey Tracking table in the Survey 

Tracking database with the date ERG received 

the returned questionnaire. 

 Perform a completeness check using a checklist 

to ensure that all applicable files are returned and 

completed. 

 Report to EPA the number of questionnaire 

responses that have been received to date (weekly 

or as directed to do so by EPA). 

 Create a working folder for each respondent that 

has submitted a questionnaire response. Save a 

working copy of questionnaire responses and 

attachments in the working folder for 

questionnaire review. 

 Save all data review and follow-up 

documentation in the working folder for each 

respondent. 

 Periodically deliver batches of the economic and 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 62 of 109 

financial part of the questionnaire to EPA’s financial contractor, if such sections of the 

questionnaires are received by ERG. 

 Store the returned electronic media and any paper documents in the ERG project archive as 

original files. These files will not be altered or changed, regardless of any issues flagged during 

data review. 

Data Review. ERG will develop Data Review Guidelines to ensure the quality and completeness of the 

questionnaire responses. The data review guidelines include the data review procedure, the technical 

review checklist (which is unique to each questionnaire and is based on the survey questions asked), and 

the documentation required for questionnaire review and any resulting respondent follow-up. Attachment 

F presents an example Data Review Guideline from the Airport Deicing Project. Included within the Data 

Review Guideline of Appendix F is an example technical review checklist. 

The data review procedure and technical review checklist include both consistency checks and 

reasonableness checks and examples of each of these include: 

Consistency Checks 

 Check that information from part to part, and section to section, throughout the questionnaire is 

consistent (e.g., the same number of process and treatment units are reported throughout); 

 Check that ID codes used throughout the questionnaire are not repeated or have multiple sets of 

information; and 

 Check that the information submitted on block diagrams or flow balances correspond to the 

information provided in question responses. 

Reasonableness Checks 

 Check all applicable questions are within specified logic ranges; 

 Check that any calculations performed by the respondent are correct; and 

 Check that all information provided for one system or unit logically makes sense (i.e., the influent 

and effluent match for a pond system). 

ERG will use the following steps when conducting questionnaire review. 

1. The first technical reviewer performs a technical data review of the questionnaire response, using 

the technical review checklist. Completion of each check will be marked by the first technical 

reviewer’s initials in the technical checklist. 

2. A second technical reviewer performs a QA review of a specified percentage of the 

questionnaires to ensure the completeness and accuracy of the primary review. ERG, in 

consultation with the EPA WAM, will determine the percentage of QA review to conduct and 

will document the percentage in the Data Review Guidelines. Completion of the check will be 

marked by the second technical reviewer’s initials in the technical review checklist. 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 63 of 109 

3. The first and second technical reviewers update the review status of the questionnaire in the 

Survey Review table in the industry tracking database after completing questionnaire review. 

ERG will periodically report the number of reviewed questionnaires to EPA. 

4. After the first technical review is complete, first reviewers will contact facilities to discuss any 

follow-up questions regarding responses determined to be incomplete or inconsistent. The first 

reviewer will also contact facilities regarding answers considered illogical or unreasonable 

(according to reasonable ranges identified by ERG for each question). Reasonable ranges, 

included in the technical review checklist, are determined by ERG in consultation with the EPA 

WAM using information from industry studies, site visits, facility contact information, and/or 

existing datasets. 

Any outstanding issues regarding the technical review checklist will be documented in the working file 

for each facility. Outstanding issues that require respondent follow-up will be documented according to 

the data review guidelines and saved in the working file for each facility. Every telephone call and email 

contact with the respondent will be documented in the Follow-up Questions tab in the technical review 

checklist. If a respondent requests follow-up conversations in writing, ERG will prepare and document all 

follow-up letters and responses. Once all issues are resolved, ERG will correct the working copy of the 

questionnaire. 

During the review and follow-up process, ERG will make changes to the working copy of the 

questionnaire. The first reviewer will document every potential change to the questionnaire that results 

from questionnaire review and facility follow-up in the Change Form tab of the technical checklist. This 

documentation will include the question number, cell reference (if EPA used an MS Excel® 

questionnaire), a comment explaining the change, the date changed, and the initials of the reviewer who 

changed the document. During the QA review, the second reviewer will confirm these changes to ensure 

that they are complete and appropriate. Once the second review is complete, the first reviewer will make 

changes to the working copy of the questionnaire. ERG is making changes to the working copies of the 

questionnaire to minimize the number of changes made to the data once extracted into the database. All 

technical checklists will be saved in the working folders of each facility and ultimately submitted to the 

rulemaking record. 

5.3.2 Questionnaire Data Acceptance Criteria 

The criteria ERG will use to evaluate the quality of data collected in a questionnaire are presented in 

Table 5-1, along with acceptance specifications for each criterion. ERG will follow the procedures 

outlined in the data review guidelines for any questionnaire responses that do not meet the following 

acceptance criteria. 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 64 of 109 

Table 5-1. Industry Questionnaire Data Acceptance Criteria 

Questionnaire Data 

Acceptance Criterion Description/Definition Data Acceptance Specification 

Unambiguous For an individual response, can the 

response to the question be 

interpreted without confusion? 

 Units of measure are identified. 

 Responses to open-ended questions are clear. 

The response indicates that the question was 

interpreted consistently by multiple respondents.

Internal consistency For an individual response, are the 

answers to one question consistent 

with answers to other questions? 

 Responses follow questionnaire skip patterns. 

 Responses to multiple questions asking for 

similar or related information are comparable. 

Completeness (Facility 

Response) 

For a facility’s response, were all 

applicable questions answered? 

A facility’s response must include answers to all 

applicable questions. The EPA WAM may specify 

that less than complete answers are acceptable on a 

facility-specific basis. 

Completeness (Question 

Response) 

For each question, was it answered 

by all respondents? 

Each question must be answered by all respondents 

to which it applies. The EPA WAM may specify 

that less than complete answers are acceptable for 

specific question responses. 

Reasonableness For an individual response, are 

reported values consistent with 

operating procedures at the 

industry of interest? 

A facility’s response will be checked against logical 

responses provided in the technical data review 

checklist. ERG will follow up with any respondent 

whose responses appear to be outliers. 

Representativeness Did EPA receive sufficient 

response to the questionnaire to 

reflect the population of the 

industry of interest? 

EPA’s goal is to receive at least 90% response rate 

in each sample frame. EPA’s statistical support will 

determine if the number of responses received 

represent each sample frame. 

Comparability Is the information provided in 

response to the questionnaire 

consistent with facility 

descriptions available from other 

sources, detailed study, and/or site 

visit data? 

Are unit operations reported the same as data 

collected from other sources, studies, or site visit 

data? 

Because EPA’s surveys are conducted under the authority of Section 308 of the CWA (Federal Water 

Pollution Control Act, 33 U.S.C. Section 1318) and response is mandatory, ERG anticipates high 

response rates and high completeness rates. ERG will determine completeness by tracking when, and how 

many, questionnaire responses are reported in the industry tracking database. In addition, ERG will report 

to EPA any instances of completeness less than 100 percent and will follow up with respondents to ensure 

complete responses by all respondents. ERG will also document any instances of respondents not being 

required to complete a questionnaire due to their current operating conditions or because they are granted 

a response waiver by EPA. 

5.3.3 Site Visit Primary Data Review and Acceptance Criteria 

Site visit data are subject to the same data review and acceptance criteria as survey data and the 

acceptance criteria displayed in Table 5-1 apply. 
Revision No. 1 

Date: October 2013 

Section No. 5 

Page 65 of 109 

5.3.4 Achieving the Contract’s Overall Quality Objectives 

By its nature, primary data collection through questionnaires or site visit surveys will reflect the 

information and view of the facility providing the data and thus will represent some facility bias. EPA and 

ERG work to achieve a full picture of the industry by developing a representative sample frame for 

questionnaire efforts and by developing survey questions that solicit information that is factual. When 

“open ended” questions that may solicit opinions are required, ERG strives to develop a clear context for 

the response. ERG’s survey data handling procedures described in Sections 5.1.3 and 5.1.4, along with 

ERG’s quality control procedures described in Section 5.3.1, ensure proper handling of survey data and 

review of those data to maintain its integrity. In addition, these procedures also help to ensure that survey 

data is not improperly altered during the collection and/or analysis process. EPA’s main goal in collecting 

primary data through questionnaires and site visit surveys is to identify information that is specific to an 

industry and/or topic of interest and thus usable for EPA’s purposes. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 66 of 109 

6.0 GENERATION OF PRIMARY DATA THROUGH SAMPLING AND ANALYSIS

Under Contract EP-C-12-021, ERG may be directed by EPA to conduct sampling and analysis in support 

of industry studies and/or ELG development. The objectives of collecting primary sampling data are to 

characterize specific wastestreams, pollutants, concentrations, and flows; evaluate treatment performance 

of technologies and practices; and otherwise assess industry circumstances that answer the questions 

posed in Table 2-2 of Section 2.2. 

This section describes ERG’s data gathering process using field sampling episodes or sampling programs, 

ERG procedures for management of sampling data, and QA/QC procedures that apply to sampling and 

analysis activities. Since sampling program design is industry- and site-dependent, ERG develops a 

Sampling and Analysis Plan (SAP) for each sampling episode to fully define the type of sampling to be 

conducted, matrices to be sampled, analytes to be measured, field data to be collected, and sampling and 

analytical methods ERG will use to complete the sampling episode. Each measurement parameter in an 

EAD sampling episode is potentially critical; however, any measurements that are needed for information 

only may be noted in the SAP. Any deviations from the QA/QC procedures outlined in this section should 

be documented in the SAP and/or in a SQAPP as directed by the EPA WAM. Non-standard situations, 

not covered by this PQAPP, would also be documented in a SQAPP. 

Generally, ERG transfers samples collected during a field sampling episode or sampling program to labs 

specified by EPA’s Sample Control Center (SCC). In these cases, SCC contracts with the specific 

laboratories and conducts review of the laboratory results generating data review narratives. It is also 

possible that as part of a wastewater sampling program, ERG may be directed by EPA to arrange for 

analysis of samples by appropriate analytical laboratories. In these cases, ERG will be responsible for 

ensuring that the laboratories analyze samples using correct methods as specified in their subcontracts and 

that they are following all required QA/QC procedures. ERG may also be directed by EPA to review the 

analytical data generated by the laboratories and prepare the data review narratives. In either case, the 

laboratory contracts will specify the laboratory analyses to be performed, the analytical methods, and the 

required laboratory turn-around-times for analytical results. ERG notes that laboratory turn-around-times 

are specified in each laboratory contract and may vary. EPA will pay higher costs for quicker turnaround-times which should be considered by EPA when specifying this criterion. 

EPA’s Policy to Assure Competency of Laboratories, Field Sampling, and Other Organizations 

Generating Environmental Measurement Data under Agency-funded Acquisitions

(http://epa.gov/fem/pdfs/fem-lab-competency-policy.pdf) requires that: 

“organizations (e.g., laboratories, field sampling and measurement) generating 

environmental data through measurement under Agency-funded acquisitions must submit 

documentation of their competency... in the fields of analyses to be conducted prior to 

performing such analysis.” 

Prior to subcontracting with or otherwise procuring services from an organization to perform any field 

sampling, field measurements, or laboratory analyses under this contract, ERG will request and evaluate 

appropriate documentation to verify that the organization is competent to perform the applicable field or 

laboratory services, in accordance with EPA’s organizational competency policy. ERG will retain records 

of that evaluation as part of ERG’s files and make the evaluation records available for inspection by the 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 67 of 109 

EPA Project Officer or designee upon request. ERG will retain the evaluation records for the duration of 

this contract. 

6.1 Field Sampling

For sampling programs conducted under Contract EP-C-12-021, ERG will support EPA in designing the 

sampling program, developing site-specific sample plans, collecting the samples, shipping them to the 

laboratories, and analyzing the sample results. ERG will collect engineering data during the sampling 

episodes to characterize production and flow during sample collection. ERG will review these data for 

completeness (i.e., that data are collected for each day and each sample point) and comparability (i.e., that 

data compare to production and flow data reported in any plant-specific documentation). 

If directed by EPA, ERG will also support EPA in designing industry self-monitoring programs, ensuring 

samples are collected properly, shipped to the laboratories, arrive at the laboratories on time and in good 

condition, and are analyzed. ERG will also ensure that plants that are part of a self-monitoring program 

will provide the engineering data associated with the industry self-monitoring program samples. ERG will 

also review these engineering data for completeness and comparability. 

6.1.1 Sampling Process Design 

ERG will assist EPA in conducting single or multiple-day sampling episodes and obtaining monitoring 

data from facilities of interest to the industry study or ELG development project. ERG will support EPA 

in selecting plants for sampling based on the plant’s specific operating characteristics. The types of 

operating characteristics considered for selecting plants for sampling include: 

 Type of industrial operation; 

 Plant location, age, and/or size; 

 Types of production operations and their wastewater discharges; 

 Type and age of wastewater treatment and control systems; 

 Types of other control operations impacting air and solid waste generation; 

 Plant pollution prevention operations/procedures; and 

 Viable sampling locations. 

ERG will develop a site-specific SAP and/or a generic SAP. A generic SAP applies to a sequence of 

sampling episodes that are covered under one program and will have a similar sampling methodology and 

objective. The generic SAP may then be supplemented with site-specific SAP addenda or site-specific 

SAPs. Site-specific SAPs and generic SAPs describe the planned sampling approach, including criteria 

for determining sampling point and analyte selection, sample collection procedures, specifics related to 

the planned analysis methods, field measurements and engineering data collection; sample handling and 

shipping procedures, and contact information. At the beginning of the generic or site-specific SAPs 

developed for this contract, ERG will include the rationale for the sampling design reflected in the SAP. 

ERG’s sample planning will also include a plan for protecting the health and safety of the sampling crew 

including details on location and methods to contact support services (e.g., fire and rescue, local hospital). 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 68 of 109 

Site-specific sampling plans and sampling plan addenda will clearly identify sampling points and 

corresponding list of analytes for each plant sampled. SAPs will also specify collection of process 

information, including process wastewater flow rates, chemical additions, and routine monitoring data. 

Collection of engineering data is described in ERG’s generic sampling plans and will be detailed in any 

site-specific sampling plans and addendum. 

Site-specific conditions may result in modifications to the planned sampling methodology. For example, 

if samples collected from the influent to a Flue Gas Desulfurization wastewater treatment system (at a 

Steam Electric plant) are still reacting (e.g., fizzing, bubbling) after sample collection, it may be necessary 

to leave the bottles uncapped until the reaction is complete. All deviations from a site-specific sampling 

plan will be approved by the onsite EPA representative at the sampling episode. ERG will document 

sample plan modifications (including the specifics on which samples were affected and information on 

the conditions that necessitate the modification) in both the samplers’ field notes and in site-specific 

Sample Episode Reports (SERs) that document the information and data collected during sampling 

episodes. 

6.1.2 Sampling Methods 

ERG’s SAPs describe the planned field sampling methodology for collecting samples from the various 

sample points. Each site-specific sampling plan will contain the following information: 

 Name and location of site being sampled; 

 General background information including, description of plant operations, waste streams, and 

sample point and analyte selection considerations; 

 Field measurement and engineering data collection; 

 Specific listing of sampling points and sample matrices; 

 Analytes and analytical methods; 

 Sample fractions, bottle sets, and holding times; 

 Field sampling QA/QC procedures and samples; 

 Sampling procedures including sample collection, labeling, preservation, packing and shipping 

procedures (i.e., preparation of traffic reports and information on where to ship samples); 

 Analytical laboratory contact information; and 

 Health and safety precautions and support services. 

ERG provides pre-sampling training to all of our sampling teams as described in Section 2.6. The training 

includes: 

 Review of ERG’s site-specific health and safety requirements, including personal protective 

equipment and procedures; 

 Site-specific health and safety training, if required by the plant; 

 Sampling procedures training for all crew members, including: 

- Sampling preparation; 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 69 of 109 

- Sample collection (techniques for collecting samples to be analyzed, including trace metals); 

- Preservation (chemicals, ice); 

- Packing and shipping, especially procedures for labeling and preparing bottles and packing 

coolers; 

- Field tests, including temperature and pH, sulfide, aldehydes, and oxidants; and 

 Sampling crew chief training (as needed) to expand on topics discussed in general sampling 

training. 

The remainder of this section describes ERG’s procedures for obtaining, inspecting and accepting 

sampling supplies; obtaining, calibrating, and decontaminating sampling equipment; general procedures 

for minimizing sample contamination; and procedures specific to field upset conditions that impact 

sample collection. 

6.1.2.1 Inspection/Acceptance Criteria for Supplies, Consumables, and Sampling 

Equipment 

Critical supplies and consumables are those that come in contact with samples. These include sample 

bottles, preservation chemicals, and sampling-related containers and equipment such as pickle jars, pump 

tubing, and scoops. Sampling crew chiefs are responsible for coordinating the purchase of required 

materials and the inspection of materials upon receipt to ensure they are usable for the specified sampling 

episode or program (e.g., no observable breakage, evidence of tampering or contamination). ERG will 

replace any materials or equipment that show evidence of tampering or contamination. Table 6-1 presents 

the inspection and acceptance criteria that apply to sampling supplies, consumables, and sampling 

equipment. 

Table 6-1. Inspection and Acceptance Criteria for Supplies, Consumables, and Sampling 

Equipment 

Supplies, Consumables, or 

Sampling Equipment Inspection Procedure Acceptance Criteria 

Sample bottles Check condition of bottles upon receipt 

for breakage or tampering and collect 

any cleanliness certification forms. 

Sample bottles must be unbroken and 

certified clean by the supplier. 

Preservation chemicals Check that chemicals purchased are in 

unopened containers and that no readily 

observable contamination is on the 

containers. 

Preservation chemicals must be in 

original packaging (unless safety 

requires transfer of the chemical to a 

smaller/clean container before use). 

Sampling-related materials that 

contact samples (tubing, scoops, 

pickle jars) 

Check the condition of any samplingrelated containers for breakage or 

tampering and collect cleanliness 

certification forms (if provided). Check 

sample tubing for readily observable 

contamination or wear. 

Sampling-related containers and 

equipment must be unbroken and 

clean. ERG will collect equipment 

blanks of sampling-related materials 

that contact samples to assess 

contamination. 

Non-contact sampling materials 

and/or equipment 

Check materials and sampling 

equipment to assess that they are in good 

working condition and for any evidence 

of tampering or contamination 

Sampling materials and/or equipment 

must be in good working condition. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 70 of 109 

ERG typically generates a bottle count spreadsheet that tallies the number and types of bottles required 

based on the number of sample locations, days of sampling, types of analyses, and number and type of 

QC volumes required. ERG then purchases pre-cleaned (and certified) sample bottles from Environmental 

Sampling Supply (ESS)5

 and purchases other sampling supplies and consumables from laboratory 

suppliers and retailers. ERG typically rents automatic samplers. 

For each sampling episode, ERG will determine the number of sample bottles, tubing configurations, 

bottles of blank water, zip lock bags, trash bags, automatic compositors, and other supplies needed for the 

episode based on the length of the sampling episode or program, number of sampling points, and types of 

analyses to be performed. Typical field sampling equipment that may be used for an ERG sampling 

program includes pH indicator paper, pH probes, thermometers, chlorine test kits, and sampling collection 

devices such as peristaltic pumps, automatic samplers, and bottle dippers. ERG will order extra sampling 

equipment as a contingency against breakage, malfunction, or loss. 

ERG will purchase certified clean sampling supplies whenever possible for anything that directly contacts 

samples, such as precleaned sample bottles. ERG will retain the certifications with the sampling episode 

documentation. For sampling supplies or equipment that contacts the sample that is not certified clean, 

ERG will collect an equipment blank to verify that it is not contaminated and to verify that adequate 

decontamination of sampling equipment was done by the sampling crew or supplier. Site-specific and 

generic sampling plans describe the sample collection techniques that will be used for collection of 

required blanks. 

6.1.2.2 Sampling Equipment Calibration, Maintenance, and Decontamination 

ERG will determine the specific field sampling devices to be used for a given sampling episode based on 

sampling location logistics. ERG will describe these devices in the site-specific sampling plans. 

Equipment will be dedicated to each sample point (preferable) or the crew will decontaminate it before 

reuse. 

To control sampling errors, site-specific sampling plans will contain specifications for sampling 

equipment and instructions for its use (e.g., peristaltic pumps, automatic samplers, field meters, and test 

kits). Sampling crew chiefs are responsible for ensuring that all equipment is maintained in good working 

condition and that decontamination procedures are properly followed. Prior to each sampling episode, 

ERG will calibrate the field test equipment following the manufacturer’s instructions. All variations from 

the manufacturer’s instructions will be fully documented in the sampling field logs. Spare parts for 

sampling equipment will either be supplied by the manufacturer or rental company at the time of 

purchase/rental, or ERG will order parts as needed to maintain the equipment in working condition. 

ERG may rent automatic samplers in support of EAD sampling episodes and/or programs. The rental 

company cleans the automatic samplers between uses using a household detergent. Prior to each rental, 

the rental company runs a set of diagnostic tests on the samplers and installs new internal tubing on the 

 

5

 ESS bottles are chemically cleaned using specified USEPA cleaning procedures for low level chemical analysis. 

Representative containers have been tested by independent certified laboratories for their appropriate use. ESS 

containers meet and exceed the required detection limits established by the USEPA in Specifications and Guidance 

for Contaminant-Free Sample Containers (OSWER Directive #9240.0-05A). 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 71 of 109 

sampling pump. The sampling crew will install new external tubing that will be used for the sample 

collection and the sampling crew will collect a field blank prior to the start of sampling during one day of 

the episode. Each automatic sampler will be dedicated to a specific sampling point for the duration of the 

sampling episode. ERG typically brings spare automatic sampler batteries into the field and spare 

connections and hoses as needed. If rented sampling equipment experiences operating problems, ERG 

will attempt to fix the problems or will require the rental company to supply a replacement. 

6.1.2.3 Methods for Minimizing Sample Contamination 

ERG will conduct all sampling activities in a manner that minimizes potential contamination and crosscontamination of samples. For example, if samples are collected with a peristaltic pump, ERG will install 

new tubing for each sampling point at the beginning of the episode. ERG samplers will use noncontaminating pH indicator paper or a pH meter during field tests and preservation. A separate one-liter 

glass jar will be filled during collection of each grab sample set and used for field measurements to ensure 

that thermometers and test kits do not contaminate the samples intended for chemical analysis. Sitespecific and generic sampling plans describe procedures for field measurements in more detail. 

All ERG personnel who collect samples will wear new nitrile gloves at each sampling point to avoid 

exposure to pollutants and other chemical, physical, and biological hazards, and to prevent crosscontamination of samples. Samplers will use a new pair of gloves at each sampling point each time they 

collect a grab or composite sample. Samplers will use care not to touch the insides of bottles or lids/caps 

during sampling. In addition, the collection of low-level trace metals requires special sample collection 

techniques. Site-specific and/or generic sampling plans describe sample collection techniques that will be 

used for the collection of low-level metal samples. 

6.1.2.4 Methods for Identifying Upset Conditions Requiring Suspension of Sample 

Collection 

ERG’s sampling programs are intended to characterize wastewater generation and treatment during 

normal facility operations. Thus, if process operations or wastewater treatment equipment goes down 

during sample collection and the upset is considered within the facility’s range of normal operations, ERG 

will continue the sample collection. If a process upset occurs during sampling, the sampling crew chief 

will record the times and dates for future reference during the data analysis phase. A member of the 

sampling crew will ask facility personnel for information regarding the upset and record this information 

in the field notes. In the event of an extended upset (i.e., outside of the facility’s normal range of 

operations), samplers will stop sample collection. Sample collection will not recommence until normal 

operations resume. The ERG crew chief, in consultation with the onsite EPA representative, will consider 

plant personnel’s judgment as to whether an upset is within the facility’s normal range of operations. 

6.1.3 Sample Handling and Custody 

This section describes procedures for sample handling and custody. Site-specific and/or generic sampling 

plans also contain information describing sample handling and custody. Each site-specific sampling plan 

will discuss proper sample handling in the field, including: 

 Sample collection; 

 Sample preservation and shipping; 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 72 of 109 

 Sample labeling; 

 Sample set preparation; and 

 Sample packing. 

Sampling data that EPA uses to support regulatory development programs must be legally defensible. 

Therefore, the sample must be tracked from sample collection through analysis. ERG will create sample 

labels to identify and track all samples. In addition, ERG will use field logs to document that samples 

were collected as described in the site-specific sampling plan. 

ERG will use a traffic report to maintain a record of sample collection, sample transfer between 

personnel, sample shipment, and sample receipt by the laboratory. The sampling crew will complete a 

traffic report for each cooler or box that is shipped to a laboratory. Each traffic report will contain the 

following information: 

 Sampling episode number for each plant sampled; 

 Number of containers of each sample in shipment; 

 For each sample in the shipment: 

- Sample number; 

- Collection date and time; 

- Preservation (yes or no) and pH after preservation; 

- Sample description (environmental matrix); 

- Analyses required; 

 Shipment or airbill number; 

 Date and method of shipment; and 

 The name of the person shipping the samples. 

The sampling crew chief or his/her designee is responsible for completing traffic reports, including 

referencing all applicable blank and QC samples, signing traffic reports, and noting the date and time of 

shipment. This individual will also inspect traffic reports for completeness and accuracy. Any person who 

makes changes to the form will initial the changes made. The sampling crew chief will retain one copy of 

the traffic report and two copies will accompany the sample shipment. 

If ERG does not anticipate shipping samples collected as dangerous goods, the samples will be shipped 

under the exclusion allowed for transporting laboratory samples in 40 CFR 261.4(d) and under the 

definition and exception in CFR 49 Part 173.134. For any samples that do not qualify for this exclusion, 

the field sampling crew will package and label samples for shipment in compliance with current U.S. 

Department of Transportation and International Air Transport Association dangerous goods regulations. 

After ERG ships samples to the designated analytical laboratories, ERG will follow up with either SCC or 

the laboratories (if ERG conducted the laboratory contracting) to check on sample receipt. Any sample 

loss (due to bottle breakage) will be identified by the lab and ERG will work with the laboratory and the 

EPA WAM to determine a corrective action. Typical corrective actions for bottle breakage include 

laboratory analysis using extra sample volume or ERG may be able to collect additional sample for 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 73 of 109 

analysis to use in place of the lost sample(s). ERG will document any missing or lost sample results in the 

SER. 

6.1.4 Field Data Management 

Field documentation tools for sampling under Contract EP-C-12-021 include sample labels, field logs, 

traffic reports, and flow and process data recorded on episode-specific data collection sheets completed 

by the samplers. The sampling crew chief is responsible for creating any episode-specific data collection 

sheets and reviews all field documentation during the sampling episode to ensure that all records are 

legible and complete. During the sampling episode or shortly thereafter, EPA will request any needed 

operating data from the plant engineers and will request that it be provided to ERG. ERG will confirm 

that all requested operating data are collected either during the sampling episode or received via 

mail/email. As needed, ERG will transfer or import field operating data into MS Excel® tables for use in 

the SER or placement in the sampling episode file that becomes part of the project file. 

6.1.4.1 Direct Measurements 

ERG will maintain a field log notebook and/or log sheets to document direct measurements taken at the 

time of sample collection. These direct measurements may include: 

 Sampling location and corresponding aliquot or sample collection date and time; 

 Sampling location conditions (e.g., weather conditions if outdoor sample, sample wastewater 

temperature and/or pH); 

 Sample preservation chemicals and amounts added per sample; and 

 Field measurements that may impact laboratory preparation procedures (e.g., field measurement 

of chlorine content). 

The site-specific SAP will specify direct measurements required for each sampling episode. ERG will use 

this information to provide the analytical laboratory with sample conditions that may impact the analysis 

and as supplemental information to help understand sampling circumstances if analysis results are 

unusual or unexpected. The crew chief will be responsible for ensuring that the sampling team uses and 

maintains the field log notebook and/or log sheets. At the completion of the sampling episode, the field 

log notebook and/or log sheets are stored and maintained with the project files. 

6.1.4.2 Non-Direct Measurements 

EPA’s sampling programs are usually augmented and complemented by the following types of data 

generated and provided by the plants sampled: 

 Process flow diagrams; 

 Daily process operation data; 

 Daily wastewater flow rates through the treatment systems; 

 Treatment chemical rates of use (e.g., L/min); and 

 Sludge generation (tons/day). 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 74 of 109 

ERG will also obtain additional relevant information on treatment techniques specific to the particular 

waste stream being sampled, such as retention time through the system and design and operating 

characteristics. For example, for sampling episodes at steam electric power plants, ERG/EPA gathered 

information such as coal usage, plant capacity, wastewater flow rates, sludge generation rates (if 

applicable), and retention times in the wastewater treatment process. 

ERG will develop site-specific data sheets for engineering data to be collected during the sampling 

episode. These data sheets will be included in the site-specific SAP and completed for each facility 

sampled. This information will be used to determine if the specific design or operational criteria of the 

facility’s process operations affect the wastewater characteristics. The crew chief will be responsible for 

using the data sheets and ensuring that the specified data are collected. At the completion of the sampling 

episode, any non-direct data collected is stored and maintained with the project files. 

6.1.5 Field Quality Control Measures and Goals 

Sampling QC activities are designed to measure or estimate the effect of sampling error such as sample 

contamination, effectiveness of decontamination procedures, and imprecision associated with sample 

collection. Site-specific sampling plans will specify quantities and types of blank and duplicate samples to 

be collected during each episode and ERG will work with the analysis laboratories and/or SCC (if 

applicable) to specify the additional sample volume to be collected for laboratory QC samples. The 

following subsections describe types of blank and duplicate samples that ERG’s sampling crews will 

collect during a sampling episode to assess field sampling QC. 

Field QC samples include trip blanks, bottle blanks, field blanks, equipment blanks, and field duplicate 

samples. Table 6-2 presents a list of these field QC samples, example analytes, purpose, and example 

frequency of collection. Note that not all analytical methods require all of the QC measures listed. Sitespecific sampling plans will discuss collection of these samples in greater detail. The goal for sampling 

blanks is non-detected values for each blank sample. If detected values are found in blank samples, ERG 

will work with the EPA WAM and analysis laboratory to determine the likely cause of the contamination 

and the impact of the result on the usability of the sample analyses from the sampling episode. The goal 

for field duplicates is discussed in Section 6.1.5.5. 

Table 6-2. Field Quality Control Samples 

Field Quality 

Control Example Analytes Purpose Example Frequency 

Trip Blank Volatile organics To monitor organic 

contamination of samples 

during transport, field handling, 

and storage 

One trip blank per episode for specified 

analytes. 

Bottle Blank Total and dissolved 

metals, low-level total 

and dissolved metals, 

organics 

To monitor pre-cleaned bottles 

for metals and/or organic 

contamination 

For low-level metals, one for each method 

at each sampling episode. For routine 

metals and organics, one for each method 

for the sampling program. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 75 of 109 

Table 6-2. Field Quality Control Samples 

Field Quality 

Control Example Analytes Purpose Example Frequency 

Field Blank Total and dissolved 

metals, low-level total 

and dissolved metals, 

organics 

To monitor potential 

contamination from sampling 

techniques (equipment used or 

ambient conditions) 

 For routine total and dissolved metals 

and organics, one blank will be collected 

for each type of sampling technique or 

special equipment used. At a minimum, 

one blank per episode will be collected. 

Field blanks will be collected for lowlevel total and dissolved metals at each 

sampling location and for each method. 

 For hexavalent chromium, one blank 

will be collected for each sample point. 

 For mercury, one blank will be 

collected for each sample point. 

Equipment 

Blank 

Low-level total and 

dissolved metals, 

organics 

To document adequate 

decontamination of sampling 

equipment before use by the 

sampling crew 

Once per lot of equipment cleaned (for 

each method). 

Field 

Duplicates 

All analytes except 

HEM/SGT-HEM 

To evaluate total measurement 

precision and assess potential 

sources of data variability, 

including sample collection, 

handling, preparation, and 

analysis 

One duplicate sample for every ten samples 

collected at a sampling event, excluding 

field blank samples. 

6.1.5.1 Trip Blank 

If required, the sampling crew will prepare a trip blank in ERG’s sample room before the start of the 

sampling episode by pouring an analyte-free matrix (ASTM Type 1 water) into one set of sample bottles. 

ERG will preserve the samples and transport them unopened to the facility. Trip blanks are stored at the 

sampling staging area throughout the sampling episode and then placed on ice during the last day of 

sampling and shipped unopened to the laboratory. Laboratories typically analyze these samples for 

volatile organic compounds. The goal is non-detected values for each trip blank. 

Note: EPA may elect to have a trip blank tested for other analytes (e.g., dioxins and PCBs). If EPA 

requests analysis for other analytes, appropriate sample bottles and volumes must be identified and 

specified in the site-specific SAP. 

6.1.5.2 Bottle Blank 

If required, the sampling crew will prepare a bottle blank in ERG’s sample room before the start of the 

sampling episode. The bottle blank samples are an analyte-free matrix (ASTM Type 1 water) poured into 

the appropriate sample bottles (e.g., 250-mL plastic bottle for total metals) and stored for several days 

prior to shipment to the laboratory. Laboratories typically analyze these samples for metal and/or organic 

compounds. The goal is non-detected values for any bottle blanks analyzed. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 76 of 109 

6.1.5.3 Field Blank 

The sampling crew will collect equipment field blanks for all sampling techniques used by sampling staff 

and for special equipment used during the sample collection, such as cooling coils, tubing, etc. The field 

blank evaluates potential contamination of the sample from either the equipment used or ambient 

conditions. To collect field blanks, ERG will pour or pump ASTM Type 1 water through all equipment 

used for the sample collection at the sampling location and into sample bottles. Multiple field blanks may 

be required if different sampling techniques are used or if ambient conditions are expected to vary from 

sample point to sample point. ERG will assist EPA in selecting analyses for the field blank using 

requirements of the analytical methods. Laboratories will analyze the field blank for the specified 

analytes. Site-specific sampling plans will list field blanks to be collected at each site. The goal is nondetected values for each field blank. 

6.1.5.4 Equipment Blank 

If non-dedicated sampling equipment (that contacts the samples) is used for multiple sampling episodes or 

between sampling points of a sampling episode, ERG will perform decontamination of that equipment 

before its next use. ERG will collect equipment blanks to evaluate that adequate decontamination 

occurred. To collect equipment blanks, ERG will pour or pump ASTM Type 1 water through/over the 

decontaminated equipment and into sample bottles. Multiple equipment blanks may be required if the 

equipment is decontaminated multiple times. ERG will assist EPA in selecting analyses for the equipment 

blank(s) using requirements of the analytical methods. Laboratories will analyze equipment blank(s) for 

the specified analytes. Site-specific sampling plans will list equipment blanks to be collected at each site. 

The goal is non-detected values for each equipment blank. 

6.1.5.5 Field Duplicates 

Each site-specific sampling plan will specify sample points for collection of field duplicates. ERG will 

assist EPA in selecting duplicate points based on information from each site and the goals of the project. 

EPA will collect one duplicate sample for every ten samples collected at a sampling event, excluding field 

blank samples. For example, EPA anticipates collecting four samples from each sampling point selected 

at a plant; therefore, if EPA collects samples from two sample points at a plant (total of 8 samples), only 

one duplicate sample will be collected. However, if EPA collects samples from three sample points at a 

plant (total of 12 samples), then two duplicate samples will be collected. The field blank samples 

associated with the sampling points will not be included in the count of the total samples collected. 

Field duplicate samples are collected to assess the degree to which repeated measurements give the same 

results and reflect both sampling and analysis precision. ERG will calculate the relative percent difference 

(RPD) between field duplicate results. An RPD less than or equal to 50 percent is usually considered 

acceptable; however, RPD values greater than 50 percent may also be acceptable by EPA depending on 

the purpose and intended use of the data. ERG will work with the EPA WAM to determine on a sampling 

episode/program basis what RPD values are acceptable to meet EPA’s quality goals and this acceptance 

criteria will be documented in the site-specific SAP. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 77 of 109 

6.1.6 Sampling Program Data Quality Criteria and Goals 

The goal of EAD’s sampling programs is to gather sampling data of the quality necessary to support EPA 

in evaluating industry studies and developing ELGs for EPA-specified point source categories. ERG will 

evaluate the quality of collected data using the criteria described below. Factors that ERG will consider in 

reviewing data quality include completeness, sensitivity, precision, representativeness, sample 

contamination, and comparability. The field quality control measures described in Section 6.1.5 apply to 

the precision and sample contamination quality criteria. Analytical quality control measures described in 

Section 6.3.4 apply to the precision, sensitivity, and sample contamination quality criteria. Table 6-3 

presents these data quality criteria, ERG’s sampling techniques to control them, the method of measuring 

the criteria, and desired goals for the measurement of the criteria for EAD sampling episodes and 

programs. ERG notes that Table 6-3 presents data quality goals; however, ERG’s discussion of sampling 

results with the EPA WAM may result in a determination that data are acceptable for use even in cases 

where the measurement goal is not met. ERG will document findings of the field QC review and will 

describe limitations of the data, if any, in the SERs or other project documentation directed by the EPA 

WAM prepared for each sampling episode. In general, data quality issues are identified and corrected via 

frequent communications between ERG, the EPA WAM, and the analytical laboratories. 

Table 6-3. Field Sampling and Analytical Data Quality Review Criteria 

Data Quality 

Review Criterion: 

Sampling 

Description/ 

Definition 

Sampling Techniques to 

Control this Quality 

Criterion 

How this Quality 

Criterion will be 

Measured 

Goal for this 

Measurement 

Completeness The degree to 

which the planned 

sampling program 

is accomplished. 

 Use method-specified 

preservation and type of 

bottles. 

 Ship samples to meet 

sample hold times. 

 Pack and ship samples to 

minimize or eliminate 

breakage. 

 Collect sufficient sample 

volume to account for all 

planned sample analyses. 

 Sample collection, 

shipping and storage meets 

method-specified 

temperature requirements. 

Percent of planned 

samples reaching the 

analysis laboratory 

and usable for 

analysis. 

Sufficient sample 

volume is shipped 

and received by 

the laboratory in 

sufficient time to 

analyze all planned 

sample points. 

Sensitivity The capability of a 

method or 

instrument to 

discriminate 

between small 

difference in 

analyte 

concentrations.. 

 Choice of method – some 

methods for the same 

pollutant have better 

sensitivity. 

 Collect method-specified 

volume (insufficient 

volume can result in high 

detection limits). 

Laboratory reported 

quantitation limits. 

Meet method 

specifications. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 78 of 109 

Table 6-3. Field Sampling and Analytical Data Quality Review Criteria 

Data Quality 

Review Criterion: 

Sampling 

Description/ 

Definition 

Sampling Techniques to 

Control this Quality 

Criterion 

How this Quality 

Criterion will be 

Measured 

Goal for this 

Measurement 

Precision 

(Reproducibility) 

Degree to which 

repeated samples 

and measurements 

give the same 

results. Represents 

both sampling and 

analysis precision. 

Use standard operating 

procedures for both field 

sampling procedures and 

laboratory analysis. 

Collect field 

duplicates of not less 

than 10% of study 

samples and calculate 

relative percent 

difference (RPD). 

See Section 1.1.5.5 

Laboratory analyzes 

ongoing precision 

and recovery 

standards. See 

Section 6.3.5.2 

RPD ≤ 50% or as 

determined on a 

sampling 

episode/program 

specific basis. 

OPR meets 

method 

specifications. 

Representativeness Degree to which 

sample has the 

same properties as 

other samples from 

other waste streams 

of a similar origin. 

 Select sampling points that 

will result in the most 

representative sample 

possible. Use sample 

techniques that promote 

the collection of 

representative samples 

based on sample point 

selection. 

Analytes detected 

and concentrations 

are comparable to 

those from other 

waste streams of a 

similar origin. 

The analytes and 

concentrations are 

consistent with 

engineering 

judgment of the 

waste stream 

origin. 

Sample 

Contamination 

Unintentional 

introduction of 

material into the 

sample during 

collection, 

transport, and/or 

analysis. 

 New sample tubing at each 

sampling point. 

 Use of clean gloves at each 

sampling point. 

 Conformance to clean 

hands/dirty hands 

collection methods, where 

necessary. 

 Use standard analytical 

methods, conducted using 

standard procedures. 

Blank samples (trip 

blank, bottle blank, 

field blank, and 

equipment blanks) 

will be used to 

measure 

contamination from 

sampling. See 

Sections 1.1.5.1 

through 1.1.5.4 

Laboratory blanks 

will be used to 

measure 

contamination from 

analysis. See Section 

1.3.5.4 

 No target 

analytes 

detectable in the 

blank samples 

(i.e., above the 

method 

detection limit). 

 Determination 

that any nontarget analytes 

detected in the 

blank sample 

will not impact 

use of the 

sampling results 

data. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 79 of 109 

Table 6-3. Field Sampling and Analytical Data Quality Review Criteria 

Data Quality 

Review Criterion: 

Sampling 

Description/ 

Definition 

Sampling Techniques to 

Control this Quality 

Criterion 

How this Quality 

Criterion will be 

Measured 

Goal for this 

Measurement 

Comparability High comparability 

means that data can 

be combined for 

decision making 

purposes. 

 Use standard criteria for 

selecting sampling points 

and collecting samples. 

 Use standard analytical 

methods, conducted using 

standard procedures. 

 Compare analytical 

methods and the detection 

limits they achieve to 

select the most appropriate 

methods. 

Compare waste 

characteristics 

determined through 

sampling between 

two points (either 

within a site or 

between sites) 

 Data compared 

represent the 

same type of 

wastestream. 

 Sampling points 

are equivalent. 

6.1.6.1 Completeness 

Completeness is defined in terms of percent of samples planned to be collected that are collected, and 

which the laboratory determines are acceptable for analysis. ERG will document sampling completeness 

in the SERs. 

6.1.6.2 Sensitivity 

Sensitivity generally refers to the capability of a method or analytical instrument to discriminate between 

small differences in analyte concentration. ERG will document laboratory quantitation limits (QLs) for 

non-detected sample results in its SERs. Laboratory QLs may be higher or lower than the method 

detection limit and understanding the QLs that apply to the reported data will inform EPA’s decision 

process on the use of the data. 

6.1.6.3 Precision 

Precision is a measure of agreement among repeated measurements. To evaluate precision, ERG will 

calculate the RPD among results from the field duplicates using the following equation. 

 

100

2 S D

1

S- D

RPD 





where: 

S = First sample result; and 

D = Duplicate sample result. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 80 of 109 

ERG will evaluate the relative percent difference and document this evaluation in the SER. ERG 

anticipates collecting one field duplicate sample for every 10 samples collected per episode. The results of 

the RPD analysis and the context will be examined to determine the precision of the data. 

6.1.6.4 Representativeness 

ERG’s sampling efforts measure pollutant characteristics of selected wastewaters. Thus, the sampling 

points that EPA and ERG select for each sampling episode are designed to represent particular 

wastewaters and evaluate candidate treatment technologies. In doing so, EPA must target a limited 

number of facilities having the desired characteristics. Section 6.1.1 discusses ERG’s sampling process 

design. 

6.1.6.5 Sample Contamination 

Sample contamination may cause a difference between measured value and true value. Blank samples 

(e.g., equipment blanks and bottle blanks) are collected and analyzed to estimate if there is sample 

contamination. ERG’s goal is that no target analytes are detected in these sampling blanks. Any sample 

contamination is documented and ERG will discuss the impact of any detected contamination on 

interpretation of sample measurements in ERG’s SERs. 

6.1.6.6 Comparability 

Comparability is a qualitative measure of the confidence with which datasets can be compared. Sampling 

programs conducted under this contract will achieve data comparability by following approved, standard 

field sampling procedures and by reporting results in standard units of measure required in the methods 

(analytes measured, procedures used, and detection limits achieved). In addition, the equivalence of 

sampling locations and operating characteristics will be evaluated. 

For those sampling programs that include industry self-monitoring data, ERG will identify specifics 

related to those data submitted (e.g., information on the methods represented by the data, sampling points, 

and conditions specific to the time of sampling). ERG will review data generated from the EPA sampling 

program and data generated from the industry self-monitoring program to evaluate the comparability of 

the data for each sampling point over the course of the sample collection time period. Specifically, and as 

directed by the EPA WAM, ERG will calculate the mean pollutant concentrations for pollutants of 

concern and evaluate whether the data collected from the self-monitoring program are significantly 

different from the EPA sampling program data using a t-test or other statistical test at a level of 

significance to be determined in consultation with the EPA WAM and EAD's project statistician. 

6.2 Requirements for Laboratories Participating in EAD Sampling Programs

If ERG will subcontract directly with analytical laboratories supporting a specific sampling episode or 

program, ERG will ensure that the following requirements are met: 

 The laboratory will be asked what certifications or accreditations they hold and for which 

matrices and/or methods. 

 The laboratory must document that they have run the required methods and can meet the method 

specified QC requirements. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 81 of 109 

 The laboratory must have a comprehensive laboratory QA program in place and operating at all 

times during the performance of tasks supporting the study. The laboratory must adhere closely to 

all QA/QC requirements of any SQAPP or a project-specific QAPP. The laboratory may perform 

additional QC to assure the validity and reliability of results and a description of any additional 

QC must be provided in the data package submitted to ERG. 

 The laboratory must have a comprehensive data management plan that is consistent with principles 

set forth in EPA 2185 – Good Automated Laboratory Practices, EPA Office of Administration 

and Resources Management, August 10, 1995. This data management plan will be in place and in 

use at all times during the performance of tasks under this contract. 

 The laboratory must maintain internal records in an organized manner to document adherence to 

the ERG subcontract, and these records must be available for review upon request. 

 The laboratory must perform all necessary QA/QC requirements specified in the analytical 

methods. 

 The laboratory shall perform MDL studies for all analytes based on the analytical method that 

will be used for the sampling program. Procedures and calculations for performing MDL studies 

are specified in 40 CFR Part 136, Appendix B. The laboratory must also determine a quantitation 

limit (QL) or “minimum level” for all analytes. The QLs are the lowest concentration that can be 

reliably measured within specified limits of precision and accuracy for a specific laboratory 

analytical method during routine laboratory operating conditions. The QL is the student t- factor 

times the MDL. For the seven replicates typically used in estimating MDLs, QL = 3.14 x MDL. 

 The laboratory must perform MS/MSD/laboratory duplicate analyses on samples specified by 

ERG and samples from EPA’s sampling program must be run in the same batch as these samples. 

Additionally, the laboratory must spike samples within the spiking range specified by the 

analytical method or SOP. If an MS, MSD, or laboratory duplicate fails to meet the method 

specified criteria, the laboratory must communicate the failure to ERG and work on resolving the 

issue (e.g., attempt to rerun the analysis with a higher dilution). 

6.3 Laboratory Analysis

Field sampling episodes and/or sampling programs conducted under Contract EP-C-12-021 will include 

ERG or facility shipment of samples to SCC or ERG subcontracted laboratories for analysis. The 

subsections below describe sample handling and custody procedures, laboratory instrument/equipment 

calibration requirements, identification of analytical methods, typical analytical data QC 

samples/procedures, analytical data management, and analytical data reporting. The responsibility for any 

required laboratory supplies and consumables will rest with the laboratories contracted to perform 

analysis. 

The goal of these procedures is to support the integrity of EPA’s primary data collection through 

sampling and analysis. ERG will specify any modifications or deviations from these procedures in a 

SQAPP specific to the project. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 82 of 109 

6.3.1 Laboratory Sample Handling and Custody 

When the laboratory receives samples, the laboratory sample custodian will check all bottles against the 

Traffic Reports, record the condition of the samples, and sign, date, and mark the time of receipt on the 

Traffic Reports. The laboratory sample custodian will immediately report any discrepancies or problems 

to ERG. ERG will verify the receipt and condition of all samples, including temperature and pH, if 

required by the method. Samples must be stored according to the method requirements, as well as to 

prevent degradation and contamination. 

The laboratory must have an SOP for sample custody within the laboratory. The laboratory will log all 

samples into a sample receipt logbook or computerized laboratory information system and will document 

the following information: 

 Date and time of sample receipt; 

 Project number and name; 

 Field sample number; 

 Laboratory sample number; 

 Sample matrix; 

 Analytical parameters; 

 Storage location; and 

 Log-in person’s initials. 

Typically samples are identified by a numbering system to mask location information and would not be 

handled as CBI. If, however, samples and analysis results are being held as CBI, all information relevant 

to the samples will be secured at the end of each business day and all samples will be stored in a 

designated storage refrigerator with restricted access. 

ERG will use an electronic sample tracking system to track the last date samples associated with each 

episode were received at the laboratory and the date of receipt of analytical data from the laboratories. 

The laboratory is responsible for ensuring that samples are prepared and analyzed within the required 

holding time specified by the method. 

If directed by EPA, ERG will subcontract analysis of EPA- and industry-collected samples to analytical 

laboratories. In some cases, EPA might identify analysis laboratories and ERG will ship samples to those 

locations. Following laboratory analysis of samples, ERG will, as directed, perform a review of the 

analytical data using method-specific review checklists to ensure the data are complete and reasonable. 

6.3.2 Analytical Methods 

Site-specific and generic sampling plans identify the analytes that will be measured for a sampling 

episode or program and the anticipated analytical methods that the laboratories will use in sample 

analysis. ERG will work with the EPA WAM and SCC (if appropriate) while planning any sampling 

episodes or programs conducted under this contract to determine which analytical methods should be 

specified. ERG and EPA’s determination of the analytical methods will consider the type of wastestreams 

sampled, anticipated pollutant concentrations and the potential for matrix interferences, the desired 

detection and quantitation limits, and whether the methods are EPA-approved (e.g., 40 CFR Part 136 

methods). 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 83 of 109 

For situations where ERG is coordinating laboratory analysis and EPA has chosen not to specify the 

analysis method(s), laboratories may be allowed to identify their preferred method for analysis of certain 

analytes and the quality control limits that would apply at the time they submit their bid response package 

to ERG. In these instances, ERG will work with the EPA WAM and the laboratory to review and approve 

the use of the preferred method and the analysis-specific QC limits. These QC limits may be set by the 

method, from a consensus organization compendium, or based on limits from similar EPA methods. 

Either way, the site-specific sampling plans will document the specific list of analytical methods that will 

be used for that sampling episode or program. Site-specific sampling plans will also identify the sample 

volume to be collected, the QC sample volume to be collected, the sample container type, and typical 

preservation required for each parameter. 

6.3.3 Instrument/Equipment Calibration and Frequency 

If ERG will subcontract directly with analytical laboratories supporting a specific sampling episode or 

program, ERG will confirm that laboratories selected to analyze samples are in compliance with the 

method required instrument calibrations and calibration frequencies. Instrument calibration procedures, 

including frequency specifications, tune performance checks, and maintenance requirements, are 

specified in the analytical methods. ERG’s PQAPP or project-specific QAPP will be provided to the 

laboratory upon award of the subcontract and the laboratory will be required to include a quality 

assurance statement in the data package submitted to ERG which verifies that the analytical method 

specifications have been met. In addition, the laboratory must maintain internal records in an organized 

manner to document adherence to the ERG subcontract, and ERG may review these documents as needed 

including documentation of instrument/equipment calibration checks. Whenever changes are made to a 

system or method, the laboratory must ensure that its equipment is operating properly by performing the 

appropriate QC tests. Analytical instruments requiring calibration must be uniquely identified to allow 

proper association of calibration records. To ensure that the calibration criteria are met, the laboratory 

must analyze test standards that are certified for purity, concentration, and authenticity or prepared from 

materials of known purity and composition. Laboratories must also perform preventive maintenance on 

all of their analytical equipment. Equipment maintenance log books must contain records of equipment 

usage, maintenance, modifications, calibrations, and repairs. 

6.3.4 Analytical Quality Control 

The following subsections describe the analytical QC measures that EAD typically uses for chemical 

analyses conducted under their sampling programs. Note that not all QC measures listed are required for 

all methods and the following sections are not all inclusive. Laboratories must follow the QC procedures 

required by each method. The recommended frequencies identified in subsections 6.3.4.2 through 6.3.4.5 

will apply unless otherwise specified in the SAP, an SQAPP, or a project-specific QAPP. 

6.3.4.1 Method Detection Limits (MDLs) 

An MDL is defined as the minimum concentration of a substance that can be measured and reported with 

99% confidence that the analyte concentration is greater than zero. For each method, laboratories are 

required to perform MDL studies for each instrument on which analyses will be performed and any time a 

modification is made that will affect the MDL. Laboratories must demonstrate that they obtain a MDL 

value that is less than or equal to (1) the MDL (if listed in the method), or (2) one-third the minimum 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 84 of 109 

level (ML, if listed in the method). Appendix B of 40 CFR Part 136 describes the EPA procedure used for 

establishing MDLs. This procedure requires the laboratory to analyze at least seven aliquots of a standard 

spiked at three to five times an estimated MDL. The laboratory processes the aliquots through all steps of 

the analytical method and calculates the MDL as approximately three times the standard deviation of the 

mean for the analyses. The number of aliquots analyzed determines the multiplier of the standard 

deviation. 

6.3.4.2 Ongoing Precision and Recovery (OPR) Standard 

The laboratory will prepare an ongoing precision and recovery (OPR) sample (such as an aqueous 

performance standard for metals analysis) by adding known amounts of the analytes of interest to a matrix 

with known characteristics. Reagent water will usually be used as the reference matrix. The standard will 

be processed through all sample preparation and analysis steps. Analyte recoveries are evaluated against 

method-specific limits. Note: The standard used should be from a source different than that used for the 

initial calibration standards. If a second source standard is not available, the standard should be from a 

different lot. 

Purpose: To assess laboratory performance and evaluate error sources in 

the laboratory. 

Recommended Frequency: As specified by the method or subcontract. 

QC Acceptance Criteria: As specified by the method or subcontract. 

6.3.4.3 Calibration Check Standard 

The laboratory will perform a calibration check standard by adding analytes of interest to a laboratory 

reagent blank. Analyte recoveries are evaluated using limits specified in the applicable method. 

Purpose: To ensure that the instrument calibration remains in control. 

Recommended Frequency: Every 10 analytical runs, or every 8 to 24 hours, as specified in 

the method. 

QC Acceptance Criteria: As specified by the method or subcontract. 

6.3.4.4 Laboratory Blanks 

The laboratory will analyze a laboratory blank, which consists of reagent water or “clean matrix” material 

that does not contain the analytes of interest or interfering compounds. ERG requires the laboratory to 

analyze a laboratory blank so that any sample contamination originating at the laboratory can be 

identified. 

Purpose: To ensure that the analysis system is free from interferences 

and contamination. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 85 of 109 

Recommended Frequency: Per method and subcontract requirements, which specify a per 

batch preparation frequency or an 8- to 12-hour per analysis 

frequency. 

QC Acceptance Criteria: As specified by the method or subcontract. 

6.3.4.5 Matrix Spike/Matrix Spike Duplicate/Laboratory Duplicate 

Depending on the requirements of the analytical method, the laboratory will usually perform a Matrix 

Spike (MS), and a Matrix Spike Duplicate (MSD)/or a Laboratory Duplicate. A MS is a sample prepared 

by adding a known mass of target analyte to a specified amount of matrix sample for which an 

independent estimate of target analyte concentration is available. Spiked samples are used to determine 

the effect of the matrix on a method’s recovery efficiency. 

Laboratories prepare an MS by adding known amounts of compounds of interest to the field sample. The 

spiked samples receive the same handling, extraction, cleanup, and analysis steps as the field samples. 

The laboratories then perform a percent recovery (%R) calculation: 

100

SA

(SSR SR) %R   

where: 

SSR = Spiked sample result; 

SR = Sample result; and 

SA = Spike added. 

If possible, the laboratory should determine the appropriate spiking concentration for the MS sample by 

first analyzing the original sample. ERG notes that this practice is not always conducted. In general, the 

spiking concentration should be at least twice the concentration of the analyte present in the original 

sample and ideally should be no more than 10 times that concentration. 

An MSD or a laboratory duplicate is a repeat analysis of a MS sample or an internal lab QC sample 

prepared and handled in a duplicate manner to the original sample and is used to evaluate analytical or 

measurement precision but not the precision of sampling, preservation, or storage internal to the 

laboratory. In general, organic methods will require an MS/MSD, inorganic methods will require a 

MS/laboratory duplicate, and some methods may require neither. EPA, in consultation with the laboratory 

and ERG, will review the method-specified QC sample analysis and volumes and will determine whether 

additional QC analysis is desired. ERG will document the laboratory QC requirements in the contracts 

with specific laboratories and QC sample volumes in the SAP. Typically, ERG will collect one QC 

sample volume (i.e., MS/MSD or MS/laboratory duplicate sample) for every ten samples collected at a 

sampling event, excluding field blank samples. Note specific to metals: If the MS or MSD analysis 

cannot be met using procedures described in the method (e.g., dilutions), the laboratory must perform a 

post-digestion spike on the parent sample after digestion to verify that the MS/MSD failure is due to 

matrix interferences. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 86 of 109 

Purpose: MS: To determine the effect of the matrix on a method’s 

recovery efficiency. 

Duplicate analysis: To evaluate analytical or measurement 

precision. 

Recommended Frequency: One MS/MSD or MS/laboratory duplicate sample for every 10 

samples collected at a site. 

QC Acceptance Criteria: As specified by the method or subcontract. 

6.3.5 Preparation and Analytical Batches 

The laboratory must perform preparation and analysis in batches as defined by the method. The 

appropriate QC samples must accompany the batches as specified in the method. A preparation batch 

consists of all samples prepared/extracted/digested together by the same person during the same work 

day. An analytical batch consists of samples analyzed using the same instrument within 24 hours, or a 

method specified time period. Typically, batches contain no more than 20 field samples (i.e., non-QC 

samples). 

6.3.6 Internal Standards, Labeled Compounds, Surrogates 

Internal standards may be required by the method for calibration and quantitation purposes. They are used 

to measure the relative responses of other method analytes and surrogate standards. Internal standards are 

added to the concentrated sample extract prior to analysis. When required, the internal standards must be 

added to all samples and QC samples at the same concentration. The recovery of the internal standard 

must meet the method specified criteria. 

Labeled compounds may be required by the method for calibration and quantitation purposes. The labeled 

compounds are used to correct for the recovery of the corresponding analytes. Labeled compounds are 

added to the sample prior to extraction. When required, the labeled compounds must be added to all 

samples and QC samples at the same concentration. The recovery of the labeled compounds must meet 

the method specified criteria. 

Surrogate analytes may be required by the method. They must be added to the sample before the sample 

is extracted. The purpose of the surrogate is to monitor method performance from extraction through 

analysis. When required, the surrogates must be added to all samples and QC samples at the same 

concentration. The recovery of the surrogate must meet the method specified criteria. 

6.3.7 Non-Standard EPA Wastewater Methods 

If non-standard EPA wastewater methods (i.e., those not included in 40 CFR Part 136 or those lacking 

QA/QC procedures) are anticipated for an EAD sampling episode or program, ERG will work with the 

EPA WAM and participating laboratories to determine appropriate analytical QC requirements which will 

then be specified in an SQAPP or a project-specific QAPP. Per 40 CFR Part 136.7, EPA’s guidance on 

QA/QC requirements for methods that lack QA/QC procedures includes three options. First the laboratory 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 87 of 109 

may refer to and follow the QA/QC published in the “equivalent” EPA method for the parameter, second 

the laboratory may refer to the appropriate QA/QC section(s) of an approved 40 CFR Part 136 method 

from a consensus organization compendium, or third, the laboratory may incorporate 12 QC elements into 

their documented SOP for performing compliance analyses when using an approved 40 CFR Part 136 

method that lacks such procedures. The 12 QC elements are: 

 Demonstration of capability; 

 Method detection limit (MDL); 

 Reagent blank (also referred to as a method blank); 

 Laboratory fortified blank (also referred to as a spiked blank, or laboratory control sample); 

 Matrix spike (MS), matrix spike duplicate (MSD), or laboratory fortified matrix (LFM) and LFM 

duplicate for suspected difficult matrices; 

 Internal standards (for GC/MS analyses), surrogate standards (for organic analysis), or tracers (for 

radiochemistry); 

 Calibration (initial and continuing); 

 Control charts (or other trend analyses of quality control results); 

 Corrective action (root cause analysis); 

 QC acceptance criteria; 

 Definitions of preparation and analytical batches that may drive QC frequencies; and 

 Minimum frequency for conducting all QC elements. (40 CFR 136.7). 

6.3.8 Analytical Data Management 

Analytical data management includes data review and reduction. These steps may be conducted by ERG 

(as directed by the EPA WAM) or another contractor (e.g., SCC). When ERG is not the contractor 

providing the analytical data review, ERG receives data review narratives and analytical results from 

EPA. Analytical data review and reduction steps are presented in the subsections below. 

6.3.8.1 Analytical Data Review 

Analytical laboratories will deliver electronic data packages consisting of MS Excel® spreadsheets 

containing analytical results and documentation packages consisting of traffic reports and sample receipt 

information, laboratory analysis sheets and checklists, instrument data, and analyte chromatograms, 

where applicable, to ERG or another contractor for data review. 

When directed by EPA to perform the analytical data review, ERG will develop and use method-specific 

checklists to review, verify, and validate chemical and/or biological analysis results and supporting 

documentation provided by the laboratory. (Attachment G presents an example method-specific checklist 

developed by ERG) ERG will perform this review to ensure data quality, note deficiencies, and, where 

appropriate, initiate corrective actions with the laboratories. Prompt initiation of corrective actions 

maximizes the amount of usable data generated from each sampling episode. 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 88 of 109 

The checklists provide method-specific QC objectives for each parameter in the review (e.g., acceptable 

instrument signal-to-noise ratio, use of correct spiking concentration, acceptable recovery ranges). Each 

checklist will contain the following standard sections listing the detailed criteria that ERG will use to 

verify and validate results of chemical and/or biological analyses: 

 General Review of Sample Identity and Handling – ERG will verify sample identity and 

traceability. ERG will confirm that all data required for data review, verification, and validation 

are contained in the laboratory data package and will review information in the data package to 

verify that the laboratory identified and handled samples properly; 

 Review of General QC Requirements – ERG will evaluate the common QC requirements in all 

methods that include sample preparation, digestion, and/or extraction procedures; and 

 Detailed Review of Method-Specific Parameters – ERG will perform method-specific QC 

validation, which includes reviewing the primary instrument calibrations and performance 

evaluation samples. 

After an analytical data package has been reviewed and all problems that can be corrected have been 

resolved, ERG will prepare a written data review narrative. ERG will submit data review narratives, 

completed checklists, along with a hard-copy summary data package, to EPA. The purpose of the 

narrative is to document the samples and analyses that were reviewed, summarize the criteria by which 

they were reviewed, note any deficiencies in the data, and provide the EPA WAM with recommendations 

concerning data usability. Data deficiencies will be noted in the data review narratives and reported to the 

EPA WAM. The EPA WAM will determine whether any corrective actions should be performed to 

address the deficient data, such as reanalysis by the laboratory (if sample volume is stored and available) 

and/or repeated sampling by ERG or the facility and reanalysis. As a result of data deficiency correction 

or acceptance, analytical data are typically excluded only in cases of grossly deficient quality control data 

(e.g., laboratory documentation is inadequate to evaluate quality control measures, or quality control 

samples results differ significantly from method requirements). 

ERG will use standard language to communicate problems and recommendations. Table 6-4 lists potential 

data limitations and associated qualifiers that may be used in EPA’s sampling programs. Any data 

qualifiers will be fully described in the final laboratory packages and data review narratives. 

Table 6-4. Data Quality Qualifiers for Sampling Program 

Potential Data Limitations Associated Qualifiers 

Example Data 

Qualifier Flag 

Holding time exceeded (for analytes subject to loss or 

degradation). 

Sample analyzed outside holding 

time. 

H 

OPR outside of the performance specifications. Data are of acceptable quality but 

may reflect either minimum or 

maximum values 

O 

MS/MSD recoveries above or below performance 

specifications. 

Data are of acceptable quality but 

may reflect either minimum or 

maximum values. 

R 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 89 of 109 

Table 6-4. Data Quality Qualifiers for Sampling Program 

Potential Data Limitations Associated Qualifiers 

Example Data 

Qualifier Flag 

Surrogate recoveries above or below performance 

specifications.

Data are of acceptable quality but 

may reflect either minimum or 

maximum values. 

S 

Sample results greater than five times but less than ten 

times blank result. 

Data may reflect maximum value L 

Sample result greater than upper end of calibration range. Include the value corresponding to 

the upper limit of the calibration 

range in the database with a “>” 

sign. 

E 

Sample result less than five times blank result. Include the value in the database as 

not detected at the adjusted method 

detection limit. 

ND 

Duplicate or MS/MSD relative percent difference greater 

than specified limit. 

Data provided represent the best 

obtainable data. 

T 

Grossly deficient quality control data. Exclude data from database (with 

EPA WAM concurrence). 

Data Qualifier 

Flag Not 

Applicable 

6.3.8.2 Analytical Data Reduction 

Each laboratory assigned to analyze wastewater samples under this contract will ensure accurate and 

contract-compliant data prior to release of the data. The laboratory will cross-reference all of its 

laboratory QC data to sampling episode-specific analyses. Documentation of the laboratory QC analyses 

is included in the laboratory data package. ERG will take the analysis results from the laboratories 

(reported by sample number) and correlate them to the appropriate trip blank, field blank, equipment 

blank, method blank, field duplicate, MS, MSD, and/or field duplicate analysis in the SER. All dates 

pertinent to the sample episode (e.g., dates on which each sample is collected, shipped, received by 

laboratory, prepared, and analyzed) will be referenced to the appropriate analytical holding times. 

The analytical laboratories will perform the analytical data reduction. The laboratories will monitor data 

reduction using results produced for internal standards, surrogates, laboratory duplicates, and spikes. 

The laboratories will each perform quantitation according to protocols specified in laboratory 

subcontracts in one of the following ways: 

 By using the isotope dilution technique; 

 By using the internal standard technique; 

 By using the external standard technique; 

 By using known response factors specified in the methods; 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 90 of 109 

 By comparing the instrument response to a built-in scale; 

 By titration to a reaction-defined endpoint; or 

 By comparing instrument response from a treated sample to a calibration curve generated from 

instrument responses to known concentration standards. 

Results will be reported in units of weight/volume (such as micrograms per liter (μg/L) or milligrams per 

liter (mg/L)) for aqueous samples, in units of weight/weight (on a dry-weight basis) for solid samples, or 

in other units as specified by the method or the laboratory subcontract. Laboratory data results are 

provided either to EPA (who will then forward them to ERG) or to ERG directly. Section 6.3.6 describes 

ERG’s procedures for putting laboratory results into a database format for presentation and analysis. 

6.3.9 Analytical Data Reporting 

Upon completion of data review, reduction, and validation procedures for all samples and analytes 

included in a sampling episode, ERG will prepare a database that summarizes analytical results in 

accordance with the database development procedures described in Section 8.0. To increase efficiency 

among data users, ERG uses a standard database format for all industry studies. Table 6-5 shows the 

typical structure of database files and source of data. The structure of the database files may be reviewed 

and changed as appropriate for a specific program. 

Table 6-5. Typical Analytical Database Structure 

Field Description Field Name Field Length Source of Data 

Analyte Concentration AMOUNT 16 Laboratory 

Analyte Name ANALYTE 75 Laboratory 

CAS Registry Number CAS_NO 10 Laboratory 

Calculated Detection Limit DETLIMIT 8 Laboratory 

Dilution Factor DILUTION 8 Laboratory 

Episode Number EPISODE 4 Upper Right Corner of Traffic Report 

Laboratory Name LABNAME 5 Laboratory 

Column or Extraction Type PROCEDUR 10 Laboratory 

Reason for Qualification REASON 60 Laboratory 

Sample Number SAMPLE 5 Left Column of Traffic Report 

Data Qualifier Flag QUAL 11 ERG/Other Contractor Data Review 

Concentration Units UNIT 7 Laboratory 

Laboratory Method Used METHOD 12 Laboratory 

To communicate the quality of individual results, ERG chemists (when ERG is conducting the analytical 

data review) or other data review contractors will represent the data qualifier information as flags that 

indicate the result of the data review. Examples of typical data qualifier flags are shown in Table 6-4. 

ERG will retain all sample collection records in the files for the project. These records will include sitespecific sampling plans, field sampling log sheets, field data sheets, preservation log sheets, and traffic 
Revision No. 1 

Date: October 2013 

Section No. 6 

Page 91 of 109 

report forms. Copies of these records will be provided to EPA as requested. Laboratory result packages 

(and laboratory data review packages if ERG is conducting the data review) will also be retained with the 

project files. 

At the completion of each sample episode, ERG prepares a draft SER that includes descriptions of the 

facility, wastewater sources, wastewater treatment, sampling points and sample collection, and deviations 

from site-specific and generic sampling plans. After receipt of all analytical results (including the data 

narratives), ERG will prepare a final SER for each sampling episode. The final report will provide 

descriptions of the facility, wastewater sources, wastewater treatment, sampling points and sample 

collection, deviations from site-specific and generic sampling plans, summary tables of the analytical 

results, and discussion of the quality control results (including field, equipment, and trip blanks; sample 

duplicates; and the results of the MS/MSD). The report will also present in-field measurement data and 

engineering data provided by the facility. 

As noted in Section 6.3.8.1, any data deficiencies are identified and discussed in the data review 

narratives, including any data excluded from the analytical database due to grossly deficient quality 

control data. The SER will not include excluded data and it is the EPA WAM who decides whether other 

flagged data are usable for the purposes of the sampling program. The data review narratives are 

maintained as part of the project record and are included as an attachment to the final SER if directed by 

the EPA WAM. 

Information collected through sampling episodes claimed as CBI by sampled facilities will be handled 

according to the procedures outlined in ERG’s CBI security plan entitled Security Plan for Handling 

Confidential Business Information Under the Clean Water Act Contract EP-C-12-021, dated November 

9, 2012 and approved by Ahmar Siddiqui, EPA’s Document Control Officer, on January 9, 2013. Only 

staff demonstrating their understanding of CBI handling procedures by passing a written test will be 

permitted to access CBI. 

6.4 Achieving the Contract’s Overall Quality Objectives

ERG’s field sampling and QC procedures described in Section 6.1.5 and analytical QC procedures 

described in Section 6.3.4 are specifically designed to provide EPA with analytical data of known quality 

and integrity. EPA’s main goal in collecting primary data through sampling and analysis is to identify 

information that is specific to an industry and/or technology of interest and thus usable for EPA’s 

purposes. 
Revision No. 1 

Date: October 2013 

Section No. 7 

Page 92 of 109 

7.0 GENERATION, EVALUATION, AND/OR ANALYSIS OF DATA USING SPREADSHEETS

ERG uses both existing data and primary data collected or generated under Contract EP-C-12-021 to 

conduct technical evaluation and analysis in support of EAD’s industry studies and ELG development. In 

support of those efforts, ERG will often use spreadsheets to collate, compare, calculate, and analyze data. 

Because use of spreadsheet tools (usually MS Excel®) is integral to much of our work, ERG has 

developed standard procedures for developing, documenting, and evaluating the accuracy of spreadsheet 

data with the purpose of providing EPA with data of known quality and integrity. This section discusses 

ERG’s QA/QC procedures specific to spreadsheet development and use along with the staff responsible 

for each of these steps. 

7.1 Developing Spreadsheets

A spreadsheet is an electronic table that is used to enter, arrange, calculate, and analyze data. ERG uses 

spreadsheets to store and manipulate data as well as to present data in a report or table format. Planning 

for spreadsheet development usually occurs in coordination with the EPA WAM and should reflect the 

overall technical approach that the EPA WAM and ERG WAM have outlined for the task. A spreadsheet 

developer is the person who creates, names, and uses the spreadsheet; it is the developer’s responsibility 

to ensure that data in the spreadsheet are imported or entered correctly, clearly identified and labeled, 

properly referenced, and that mathematical functions used for manipulating data are executed accurately. 

ERG maintains version control for spreadsheets by including the date or version number in the file name. 

7.2 Quality Control and Data Quality Goal

All project spreadsheets undergo technical review by technical/project staff other than the spreadsheet 

developer. The responsibilities of the technical reviewer are to verify that the technical approach and 

procedures used in the spreadsheet are reasonable and logical, verify that the spreadsheet documentation 

is complete and clear, and verify that calculations and results are accurate by manually verifying equation 

cells. 

The ERG WAM will work with the EPA WAM, project staff developing a spreadsheet, and the technical 

reviewer to determine the level of QC necessary (i.e., whether all data points and calculations must be 

checked or whether only a percentage should be checked). For example, a spreadsheet column may have a 

repeating calculation that runs the length of the column rows and a check of the calculation for one row 

will serve to ensure accuracy for all rows. Depending on the spreadsheet size and work 

assignment/project budget and time constraints, it may be appropriate for ERG to initially check only a 

percentage of large spreadsheets. If, through the percentage check, spreadsheet corrections are identified, 

ERG will address corrections for those columns or rows identified and any others that may be impacted 

by the same issue. ERG will work with the EPA WAM to determine whether expanded QC checks are 

necessary. The technical reviewer will identify any errors or deficiencies in data labeling and references 

to the spreadsheet developer who will implement the needed corrections. 

ERG’s data quality goal is for 100 percent accuracy of the data in the spreadsheets developed on this 

contract. ERG’s spreadsheet review is designed to identify and correct any data errors to achieve this 

goal. 
Revision No. 1 

Date: October 2013 

Section No. 7 

Page 93 of 109 

7.3 Documenting Spreadsheet Quality Control

ERG staff will document technical review of spreadsheets using an electronic Deliverable Sign-Off Sheet 

(see Attachment H) or QA worksheet. As deemed necessary, the technical reviewer will provide a written 

summary of the data checked, errors or problems found, and recommendations for revisions. This 

summary may be provided separately to the spreadsheet developer or included in a QA worksheet in the 

spreadsheet workbook. 

7.4 Identification and Control of CBI in Spreadsheets

The spreadsheet developer is responsible for determining if any information used in the spreadsheet is 

CBI. If so, s/he follows ERG’s CWA CBI Security Plan procedures during spreadsheet development, 

review, and distribution. All electronic spreadsheet files containing CBI are stored on ERG’s CBI server 

in dedicated and access-controlled project areas. CBI spreadsheet files are identified as containing CBI by 

including “CBI” in the file name and including an electronic CBI cover sheet at the front of the 

spreadsheet workbook. 

7.5 Quality Assurance

It is the responsibility of the ERG WAM and Project QA Coordinator to require and check that ERG’s 

spreadsheet development procedures described in this section are followed. The ERG WAM will specify 

the necessary quality control for spreadsheet development as part of upfront task planning. In addition, 

deliverable review records or QA worksheets provide documentation for the EPA WAM that spreadsheet 

QC occurred. The Project QA Coordinator will support project staff in the use of the spreadsheet SOP and 

will be able to verify that QC is occurring when they audit project quality procedures by reviewing QC 

documentation. 
Revision No. 1 

Date: October 2013 

Section No. 8 

Page 94 of 109 

8.0 GENERATION, EVALUATION, AND/OR ANALYSIS OF DATA USING DATABASES

ERG uses both existing data and primary data collected or generated under Contract EP-C-12-021 to 

conduct technical evaluation and analysis in support of EPA’s industry studies and ELG development. In 

support of those efforts, ERG will often use databases to store, organize, calculate, and analyze data. ERG 

uses databases to store and analyze questionnaire responses, sampling results, long-term treatment 

performance data, and for development of cost, loading, and environmental impact calculation models. 

Because use of database tools (usually MS Access®) is integral to much of our work, ERG has standard 

procedures for developing, documenting, and evaluating databases and database queries with the purpose 

of providing EPA with data of known quality and integrity.. This section discusses quality control and 

quality assurance procedures specific to databases and database queries along with the staff responsible 

for each of these steps. 

8.1 Developing Databases

A database is a collection of data that is organized so that its contents can easily be accessed, managed, 

and updated. The most prevalent type of database is the relational database, which is a tabular database in 

which data are defined so that they can be easily reorganized and accessed for data calculations, with a 

shared attribute providing a relationship between data tables. Planning for database development will 

occur during the planning stages of a task. Examples include database development for a 

questionnaire/survey effort and when planning an approach for developing industry costs and loads to 

evaluate baseline and regulatory option loads, load removals, and industry costs. The ERG WAM will 

work in close coordination with the EPA WAM when planning these activities as part of the systematic 

planning process for the task and is responsible for reviewing and approving the written plan for overall 

development and quality control of the database(s). 

The ERG WAM will identify a database task leader responsible for the database development and 

analysis effort. In some cases, the task leader will work with a database developer or the task leader will 

also be the developer. The database developer is responsible for selecting the appropriate database 

software, describing the database development plan, and working with the task leader to assign staff to 

review, data enter, and QC project data that are included in the database. Data entry for databases may 

include manual input of data into forms or tables or importing of datasets into the database. The task lead 

will also be responsible for ensuring that all supporting documentation for the information in the database 

is referenced and that documentation of the final database is prepared. ERG maintains version control for 

databases by including the date or version number in the file name. 

8.2 Developing Database Queries

Database queries are used to view, change, and analyze data contained in one or more data tables. They 

are constructed either directly using structured query language (SQL) or indirectly with a user-friendly 

interface. There are two types of queries: select and action. Select queries retrieve a result from a set of 

records in one or more tables. Action queries ask for additional operations on the data, including creating 

new data, adding to existing tables, updating data, and/or deleting data. In addition to database queries, 

database macros may also be used as a shortcut to task activities that are performed repeatedly. Examples 

include opening a report, running a query or set of queries, or closing the database. Another common 

feature of databases are database modules, which are a collection of declarations, statements, and 
Revision No. 1 

Date: October 2013 

Section No. 8 

Page 95 of 109 

procedures that are stored together as a unit, similar to a macro. ERG constructs database modules using 

Visual Basic for Applications (VBA) programming language, which can allow for a more complex 

combination of procedures to retrieve, change, or manipulate data. A module can be either a class module 

or a standard module. Class modules are attached to forms or reports and contain procedures that are 

specific to the form or report. Standard modules contain general procedures that are not associated with 

any other object. 

A query developer is a person who creates, names, and uses database queries, macros, or modules to 

develop results specified by the database task lead and the database plan. The query developer designs the 

desired actions and mathematical functions, creates a query documentation table in the database, ensures 

that their queries have been reviewed, and addresses any comments from technical review. 

8.3 Quality Control and Data Quality Goal

All project databases and database queries undergo technical review by staff familiar with database 

programming. The responsibilities of the technical reviewer are to verify that the technical approach and 

database design structure and queries are reasonable and logical; verify that original data are input 

correctly; verify that queries, calculations, and results are accurate; verify that sequential queries are 

working properly; and check that database documentation is complete and clear. The ERG WAM will 

work with database task leader and technical reviewer to determine the level of QC necessary. If needed, 

the Project QA Coordinator should be consulted for guidance in determining effective ways to conduct 

the QC. The Project QA Coordinator will audit database development to ensure that all required reviews 

have been conducted and documented. 

ERG’s data quality goal is for 100 percent accuracy of the data put into or generated by the databases 

developed on this contract. ERG’s database and database query reviews are designed to identify and 

correct any data errors to achieve this goal. 

8.3.1 QC for Database Data Entry 

Procedures for ensuring quality control in data entry include the following: 

 Plan a consistent method for data entry. 

 When manually entering data, use double key entry where possible. This method allows for 

comparison of the two entries to identify errors and correct them. 

 When importing data, check that data fields have been successfully imported, ensure that zero and 

blank values are handled appropriately, and ensure that data truncation has not occurred due to 

setting field sizes too small or because the correct field type was not selected. 

 Count records to ensure the number of records in each table is correct. 

 For data entry input forms, have another team member or programmer review the programming 

code to ensure that data are saved in the correct tables and formats. 

 Document any data entry reviews in the database documentation table. 
Revision No. 1 

Date: October 2013 

Section No. 8 

Page 96 of 109 

8.3.2 QC for Database Queries and Outputs 

Procedures for ensuring quality control in database queries and database outputs include the following: 

 Verify record counts; 

 Check equations for accuracy by performing hand calculations to compare to outputs; 

 Confirm that constants are correct; 

 Check that items are joined appropriately; and 

 Manually check records. 

8.4 Documenting Database Quality Control

ERG staff will document technical review of databases and their underlying database queries. QC 

documentation may be maintained in the database itself (as part of the database documentation table) or in 

separate documentation tables (usually MS Excel® files). Database QC documentation will include 

resolution of double-key entry comparisons (for manual data entry), database review revisions and 

corrections, and documentation of QC for queries. 

8.5 Identification and Control of CBI in Databases

The database task leader and database developer are responsible for determining if any information used 

in a database is CBI and, if so, following ERG’s CWA CBI Security Plan procedures during database 

development, review, and distribution. The ERG database developer will identify CBI data included in a 

database with a CBI designation tag. A typical example of this is a database composed of questionnaire 

data where some data are identified as CBI by the questionnaire respondents and some data are not. 

ERG’s questionnaires are designed with CBI designation boxes for each question, part of a question, 

and/or comment, which then provides a CBI designation tag once the data are entered or imported into a 

database. In this case, the database table containing questionnaire responses includes a column for the 

data and an adjoining column with CBI designation (yes/no). Any database containing CBI data will be 

developed, reviewed, maintained, and stored on ERG’s CBI server in dedicated and access-controlled 

project areas. Databases containing CBI data are not released to the public until and unless they are 

sanitized (i.e., all CBI data and CBI-deducible data have been removed). 

8.6 Quality Assurance

It is the responsibility of the ERG WAM and Project QA Coordinator to require and check the use of 

ERG’s database and database query quality control procedures. The ERG WAM, in consultation with the 

database task leader and Project QA Coordinator, will specify the necessary level of quality control for 

database development as part of upfront task planning. The ERG WAM will review database quality 

control documentation to ensure that QC occurred. The Project QA Coordinator will support project staff 

in use of the database quality control procedures and will verify that QC is occurring when s/he audits 

project quality procedures by reviewing QC documentation. 
Revision No. 1 

Date: October 2013 

Section No. 9 

Page 97 of 109 

9.0 GENERATION, EVALUATION, AND/OR ANALYSIS OF DATA USING MODELS

In support of industry studies and ELG development projects under Contract EP-C-12-021, ERG may 

create MS Excel® or MS Access® calculation models for: 

 Developing point source category costs, pollutant reductions, and/or non-water quality impacts 

for regulatory options under evaluation by EPA; 

 Modeling environmental impacts (such as receiving water concentrations and wildlife tissue 

values); and 

 Modeling other impacts of regulatory options under evaluation as requested by EPA. 

Additionally, ERG may identify and use existing models created by EPA, other agencies, or other entities 

to support our technical analyses conducted under this contract. This section identifies ERG’s procedures 

for generating and evaluating data using models and the quality criteria ERG will use to ensure that the 

use of these models is appropriate. 

9.1 ERG Generated Computational Models

Under this contract, ERG will create MS Access® databases or MS Excel® spreadsheets, write 

computational or module code, and create queries to develop: 

 National-scale compliance modeling tools to estimate the cost to comply with proposed 

regulatory options and estimate expected pollutant reductions and non-water quality impacts – in 

support of compliance cost and pollutant reduction modeling analyses; and 

 National-scale screening-level environmental modeling tools to estimate receiving water 

concentrations and evaluate wildlife tissue values required to meet project objectives outlined in 

Section 2.2 – in support of environmental modeling analyses. 

ERG will follow our quality control procedures for spreadsheets and databases in developing these 

models; Sections 7.0 and 8.0 provide details on ERG’s QC and assessment procedures for development of 

spreadsheets, databases, and database queries. ERG documents the following elements of our 

computational models in project-specific methodology memoranda: 

 Model description; 

 Flow charts (if necessary); 

 Database description; 

 Data sources; 

 Documentation of assumptions; 

 Discussion of any boundary or limiting conditions; 

 Documentation of default parameter values and their sources; and 

 Documentation of all equations (and sources, if applicable) used by the model to generate output. 

ERG will document code development and modifications for each version of the model. Specific 

documentation that will be kept for each version will include the following: 
Revision No. 1 

Date: October 2013 

Section No. 9 

Page 98 of 109 

 ERG’s code modules or database queries created to manipulate data will include inline 

programmer comments or documentation in a separate file describing the purpose and equations 

used in the query; 

 ERG will maintain a table (in the database, as a tab in the spreadsheet, or as a separate file) that 

documents/tracks significant changes made to the model; and 

 ERG will track errors identified and subsequent resolution in the database, as a tab in the 

spreadsheet, or as a separate file. 

ERG will assign a technical reviewer to review and verify module code and query code to check the 

correctness and accuracy of the governing equations and to assure that the database code is fully 

operational. ERG will verify that input files and changes to model input parameters are correct using our 

established technical review process as outlined in Sections 7.2 and 8.3. 

ERG will test each version of the model prior to delivery to EPA. Test cases will be created for each 

version of the model and will consist of a predetermined sequence of steps to be executed and the 

expected output or results. ERG will establish specific test case criteria and testing sequences in 

consultation with the EPA WAM. ERG will execute and document the test cases once test case criteria 

are established. 

ERG defines model calibration as the process of adjusting model parameters within physically defensible 

ranges until the resulting predictions give the best possible or desired degree of fit to the observed data. 

ERG defines model verification as the comparison of predictions between a calibrated model and 

available data that were not used in the model development and calibration. ERG will use available cost, 

loading, and stream monitoring data at select sites from secondary sources to calibrate and verify that 

model estimates are reasonable. ERG will consult with expert scientific researchers who are 

knowledgeable about model subject matter to review model output for reasonableness. The desired degree 

of fit will be determined on a case-by-case basis and in conjunction with the model verification. Model 

accuracy is usually defined in terms of percentage difference between modeled outputs to known values. 

ERG will consult with the EPA WAM on any modeling efforts to determine whether the model output 

estimates are reasonable for the intended purpose. 

9.2 Existing Models

In support of industry studies and ELG development under this contract, EPA and ERG may identify 

existing models for use that support evaluation of costs, loads, or environmental impacts. ERG’s 

procedures for assessing the quality of the data used in these models, the validity of the model calculation 

methodology, and the usefulness of modeled outputs are presented below. Before use of any existing 

model, ERG will assess the following: 

 Does the model support answering the questions posed in Section 2.2? 

 Is the origin of the model and a description of its inputs, modeling methodology, and outputs 

documented and available? 

 Has the model been evaluated and accepted for use through an EPA Peer Review process or other 

expert review process? 
Revision No. 1 

Date: October 2013 

Section No. 9 

Page 99 of 109 

 Is the intended use of the model appropriate to the point source category under review by EAD? 

ERG will consult with the EPA WAM on the answers to these questions and the EPA WAM will 

determine whether the use of the model is appropriate. 

9.2.1 Review of Data Quality in Existing Models 

An existing model will require model input as well as pre-existing model data. ERG will apply the same 

data quality acceptance criteria to existing models as those applied to existing data. Section 4.3 discusses 

ERG’s data quality criteria and acceptance criteria for existing data. 

Data quality acceptance criteria for input data used in existing models will include the following: 

 Preference will be given to primary data sources (e.g., EPA data collected directly from the point 

source category under investigation); 

 Data sources that are publicly available and generated by other federal Agencies (e.g., USGS 

databases, National Oceanic and Atmospheric Administration databases) will be considered of 

sufficient quality for use; and 

 Data sources authored by members of industry (e.g., trade association documents or reports 

authored by industry committees) or other existing data sources will be considered for use and, 

where possible, checked for comparability against other data sources. 

9.2.2 Review of Model Methodology 

ERG will utilize the following review criteria when assessing existing models for use in ELG 

development: 

Reasonableness of the Model Methodology. ERG will review model documentation to understand and 

assess the calculations and assumptions used. Models without documentation on calculation methodology 

and underlying assumptions will not be acceptable for use. ERG will discuss existing model assumptions 

with the EPA WAM to determine whether they are acceptable for the intended use. 

Model Sensitivity. ERG will test existing models using a range of input data to assess the sensitivity of the 

model to the data. ERG will discuss the results of sensitivity testing with the EPA WAM. 

Representativeness. ERG will review the intended purpose and use of existing models to assess whether 

they are appropriate for EPA’s assessment of data in support of an ELG. Models whose purpose and 

circumstances do not represent the industry, pollutants, technologies, and/or environment under study by 

EAD should not be used. 

Accuracy and Comparability of Model Outputs. ERG will evaluate the accuracy and comparability of 

modeled outputs by comparing them to existing industry data measuring or representing the same 

intended values. ERG will calculate a relative percent difference between modeled values and known 

measured values and review these differences with the EPA WAM. 
Revision No. 1 

Date: October 2013 

Section No. 9 

Page 100 of 109 

9.3 Additional Resources

ERG will develop, as directed, a project-specific QAPP to address ERG development and use of any 

environmental models not covered by Section 9.1. ERG will use EPA’s Guidance on the Development, 

Evaluation and Application of Environmental Models (CREM, 2009) when developing and applying such 

models. In addition, EPA Region 1 has developed a QAPP template specific to environmental modeling 

which can be accessed at http://www.epa.gov/region1/lab/qa/qamodeling.html. ERG will use this 

template as a basis for developing the project-specific QAPP which will detail the purpose of the model 

developed, the source and quality requirements for model inputs, the basis of the modeling methodology, 

data management procedures, and procedures for model calibration, model validation and simulation, and 

reconciliation with the user requirements. 
Revision No. 1 

Date: October 2013 

Section No. 10 

Page 101 of 109 

10.0 DELIVERABLES REVIEW

ERG will generate written deliverables as part of work conducted under Contract EP-C-12-021. Types of 

deliverables will include: 

 Work plans; 

 Quality-related documents (QMP, PQAPP, quality documentation); 

 Planning and status memoranda; 

 Technical letters or memoranda; 

 Contact documentation (e.g., telecons and emails); 

 Study reports; 

 Survey instruments; 

 Site visit reports; 

 Sampling and analysis plans; 

 Health and safety plans; 

 Sampling episode reports; 

 Technical development documents (including cost and loading reports); 

 Environmental and/or economic assessment documents; 

 Response-to-comment documents; and 

 Technical progress reports. 

This section presents an overview of QA/QC procedures that ERG uses to ensure the quality of written 

deliverables. 

10.1 Levels of Review

ERG has an established internal QC review procedure for work assignment deliverables (memoranda, 

technical and engineering reports, background documents, etc.). Table 10-1 shows the levels of review 

that ERG requires for various types of deliverables. At the direction of the EPA WAM, ERG will deviate 

from these levels of review to accommodate situations where there is limited time and/or budget. 

ERG conducts technical tasks and prepares deliverables using: 1) a conceptual review, 2) a developmental 

review, and 3) a final product review. 

A conceptual review is performed during the initial stages of work development and ensures that the final 

product and associated documentation address the needs set forth by the EPA WAM and work 

assignment. Conceptual review will be provided by senior managers knowledgeable in the work 

performed but not directly involved in the work assignment. 
Table 10-1. Levels of Review 

Work Product

Team 

Member/Technical 

Reviewer1

Conceptual 

Reviewer2

Project QA 

Coordinator

ERG 

WAM3

Internal project documentation X Planning documents/Methodology development X X X Spreadsheets and Calculations X X7Assumptions and Conclusions X X X Design of database, Data operations, and Models X X7 X Database queries and Modeling runs X Client deliveries that include data or technical analyses X X Work Plan and Cost Estimate X6 QA Project Plan (QAPP)5 X X Documentation to Client on Project Status, Financial Status, Additional Funds Required, Invoices, etc. 

 X 

Reports X X Internal project quality audit X8 

1 Review by a team member familiar with the work that did not perform the task, often the task leader. 2 The conceptual review is conducted by the Project QA Coordinator or other qualified senior staff member who is not directly involved 3 Informal emails and transmittal letters to client with no data presented will be reviewed by the ERG task lead. As needed, the ERG W

alternate reviewer to accommodate delivery schedules. 

4 Unless otherwise directed by EPA, first drafts of documents will not be reviewed by a technical editor. ERG will conduct a technical e

incorporating EPA’s comments. 

5 Meets EPA QA/R-5 requirements. 6 ERG WAM review of a work plan and cost estimate occurs when these deliverables are prepared by a team member/task leader other 7 The Project QA Coordinator checks that spreadsheet and database development QC procedures are followed by reviewing QC docum8 The Project QA Coordinator will review internal project audit reports when they are prepared by a team member other than the Projec

cases the Project QA Coordinator will conduct internal project audits for the project.
Revision No. 1 

Date: October 2013 

Section No. 10 

Page 103 of 109 

The quality of intermediate deliverables and final products is also evaluated as these work products 

evolve. This developmental review includes: 1) a check on calculations and data (see Sections 7.0 and 8.0 

for QC conducted on spreadsheets and databases), and 2) a review of draft documents to ensure that the 

direction of work is consistent with the conceptual review outline. The ongoing status of work tasks will 

also be reviewed weekly by the project team and reported to the EPA WAM. 

Final product review is conducted on all deliverables prior to delivery to EPA. 

10.2 Types of Review

For developmental and final reviews on written deliverables, ERG conducts the follow types of review: 

 Technical review; 

 Copyedit review; and 

 Technical edit review. 

ERG determines the type of document review needed based on the type of deliverable, its status (draft or 

developmental stage versus final), and the direction of the EPA WAM. Draft or developmental reviews 

will include a technical and copyedit review. All final major documents (such as Technical Development 

Documents and Final Study Reports) will undergo both technical review and technical editing. The extent 

of final deliverable editing will be dictated by the EPA WAM because it may depend on time and 

resources available as well as the desire by EPA to maintain specific text that has been developed through 

EPA internal review and should not be subject to change. 

10.2.1 Technical Review 

A technical review is conducted by senior staff with sufficient technical background and experience to 

understand and critique the work described in the technical memoranda, report, or document but who did 

not author the deliverable. The technical reviewer will assess the following: 

 The organization of the deliverable; 

 Whether the technical presentation of data is clear, logical, and sources are documented; 

 Whether any results, conclusions, and recommendations based on the technical analysis presented 

are substantiated by the data and arguments presented; and 

 Whether the writing is clear and understandable. 

A technical reviewer may also identify copyediting or substantive technical editing issues (as listed 

below). 

10.2.2 Copyediting Review 

Copyediting is generally conducted by members of ERG’s editorial team but may also be conducted by 

other project staff. A copy editor evaluates the deliverable for micro-level issues (but will note macrolevel issues for a document manager to consider). Examples of micro-level issues include: 
Revision No. 1 

Date: October 2013 

Section No. 10 

Page 104 of 109 

 Style (e.g., follows ERG or EPA style guide for the type of document); 

 Formatting (e.g., levels of headers, TOC/page numbers/graphics issues); 

 Punctuation; 

 Grammar and syntax; 

 Typos; and 

 Consistency (e.g., in use of acronyms). 

10.2.3 Technical Edit Review 

ERG’s editorial staff will also provide, as needed, substantive technical editing of final documents. 

Technical editing includes copyediting as well as evaluation of macro-level issues such as: 

 Identification of illogical thinking or poor organization; 

 Identification of difficult-to-understand material; 

 Appropriateness for the target audience or goal of the document; 

 Correction of poorly constructed sentences; 

 Review of word choice and sentence corrections to smooth awkward spots; 

 Review of passive versus active voice; and 

 Identification of redundancy. 

10.3 Quality Assurance for Deliverable Review

As described in Table 2-4, project staff authoring project deliverables are responsible for following the 

quality control procedures described in this PQAPP and any project-specific QAPP or SQAPP. The 

author of a deliverable will coordinate review and will work with the ERG WAM as needed to determine 

the appropriate levels of review and identify reviewers. The project deliverable author will then 

incorporate any needed edits or revisions based on internal review. After receipt of any EPA review 

comments on a draft deliverable, the ERG project deliverable author will again be responsible for 

addressing and incorporating any edits or revisions. 

The ERG WAM is responsible for ensuring that project staff comply with the necessary quality control 

reviews and will coordinate with the EPA WAM, when needed, to respond to any requested deviations 

from the Table 10-1 levels of review. ERG’s Project QA Coordinators assess the project for compliance 

with our stated quality control procedures through participation in project team meetings, participation in 

determining corrective action if quality issues arise, and through project audits as described in Section 

11.1. 
Revision No. 1 

Date: October 2013 

Section No. 11 

Page 105 of 109 

11.0 ERG’S QUALITY PROGRAM ASSESSMENTS AND RESPONSE ACTIONS

ERG’s quality program for Contract EP-C-12-021 includes the following main components: 

 Development of a customized QMP for the contract; 

 Development of this PQAPP; 

 Designation of a Contract QA Manager (Mary Willett) and Project QA Coordinators to support 

development of project-level QA/QC procedures to use with each work assignment; and 

 Activity-specific SOPs. 

This section discusses how ERG ensures that our QA program is implemented and ERG’s response 

actions to address quality concerns or deficiencies. 

11.1 ERG Quality Program Assessments

The responsibility for assessing the use of ERG’s quality procedures rests with all project staff as follows: 

 The ERG Contract QA Manager is responsible for ensuring that quality planning for the contract 

has been performed; she documents this planning in the customized QMP and this PQAPP. In 

addition, the ERG Contract QA Manager will ascertain that our quality control procedures are 

being implemented by participating in biweekly ERG WAM meetings where quality issues are 

discussed and through a more formal auditing of the contract projects on an annual basis using 

our QA audit checklist (see Attachment I). The ERG Contract QA Manager will review project 

audit results with the ERG WAM and Program Manager and discuss any observations relating to 

ERG procedures for future work under the contract. 

 The ERG Program Manager is responsible for supporting the ERG Contract QA Manager with 

quality planning for the contract and is also the lead in assigning responsibilities for quality 

control development and assessment at the work assignment level. The ERG Program Manager 

will address any substantive quality issues that are raised and has the authority to require 

additional quality control activities, staff training, and adjust project staffing as needed. 

 Project QA Coordinators are responsible for supporting individual work assignment and project 

teams in implementing quality control and quality assessment activities at the work assignment 

level. The Project QA Coordinator will work with the ERG WAM to support development of 

SQAPPs or individual project QAPPs as needed and will conduct project-level QA audits as 

directed by the ERG Contract QA Manager to assess implementation of QC. The Project QA 

Coordinator will complete an audit checklist and discuss audit observations with the ERG WAM 

and Contract QA Manager at the completion of any audits. 

 ERG WAMs are responsible for ensuring that project staff understand the QA/QC procedures 

they are to implement in conducting the work assignment/project. The ERG WAM will work 

with the EPA WAM and ERG’s Program Manager to decide on the amount of data checking to 

perform for spreadsheet and database development and the levels of review required for project 

deliverables. 
Revision No. 1 

Date: October 2013 

Section No. 11 

Page 106 of 109 

 ERG project staff is responsible for implementing and documenting project quality procedures as 

specified by this PQAPP, SQAPPs, or project-specific QAPPs, and for using ERG’s SOPs. 

11.2 Response Actions

Quality assurance response actions are implemented at the work assignment/project level and, when 

necessary, at the contract level as follows: 

Quality Assurance Finding Response Action Roles/Responsibilities 

Insufficient Quality Planning The ERG Program Manger and ERG 

WAM will work with the EPA WAM 

to identify and incorporate the needed 

work assignment/project quality 

planning. ERG will document QC plans 

in a systematic planning deliverable. 

The ERG Contract QA Manager and 

the ERG Program Manager will 

monitor the contract work 

assignments/projects to ensure that 

quality planning is done. 

The ERG WAM and Project QA 

Coordinators will conduct quality 

planning and document plans in a 

systematic planning deliverable. 

QC Procedures Not Followed 

(includes failure to implement a 

QC check or failure to document a 

QC check) 

The ERG reviewer or ERG WAM will 

require that project staff to implement 

the missing QC procedure(s). 

The ERG reviewer and/or ERG WAM 

identifies missing QC. 

The ERG project staff implement the 

necessary response action. 

EPA WAM identifies Quality 

Concerns 

The ERG Program Manager and ERG 

WAM will meet with the EPA WAM to 

discuss the quality concern and 

investigate the source of the problem. 

The ERG WAM will work with the 

EPA WAM to identify mitigation 

procedures and will implement fixes to 

the deliverable. 

The ERG WAM in consultation with 

EPA WAM identifies necessary QC 

and directs project staff. 

The ERG project staff implement the 

necessary response action. 

As needed, ERG Program Manager 

specifies additional or replacement 

staff and/or procedures to address the 

EPA concern. 

QA Procedures Not Followed 

(includes failure to implement QA 

review or document a QA review) 

The ERG Contract QA Manager will 

direct the appropriate Project QA 

Coordinator to implement the missing 

QA review and/or documentation. 

The ERG Contract QA Manager and 

Project QA Coordinators implement 

the missing QA and/or documentation. 
Revision No. 1 

Date: October 2013 

Section No. 12 

Page 107 of 109 

12.0 ASSESSING DATA USABILITY AND RECONCILIATION WITH USER REQUIREMENTS

12.1 Assessment of Data Usability

Components of ERG’s assessment of data usability include: 

 Data review against quality specifications and goals as defined in Sections 3.0 through 8.0 above 

or in a project-specific QAPP or SQAPP; 

 Discussion of ERG’s data quality review results in the context of the work assignment/project 

with the EPA WAM (via memoranda and/or meetings) ; 

 Decisions by EPA on the usability of the data and direction from the EPA WAM to ERG 

regarding any adjustments to the analysis (e.g., additional data identification and/or collection, 

alternative analysis, etc.) based on those decisions; and 

 Documentation of data review and usability as part of the project or regulatory rulemaking effort. 

Ultimately, it is EPA’s responsibility to make final determinations regarding data usability. ERG’s role is 

to provide EAD with recommendations regarding data usability and help reconcile data quality issues as 

necessary to help facilitate use of data by EPA to meet its needs. 

12.2 Reconciliation with User Requirements

ERG will clearly identify sources and limitations of existing data collected in all project deliverables. 

EPA WAMs will determine the usability of existing data collected under work assignments and provide 

written documentation of their determinations to ERG and the EAD QA Coordinator. ERG will 

thoroughly document all calculations and analyses for a work assignment’s project file in spreadsheets, 

databases, calculations, database programs and queries, technical memoranda, and documents. ERG will 

use only reviewed and accepted data in calculations and analyses, and any data limitations or assumptions 

will be clearly described to the user. Data produced or developed under contract work assignments will be 

reconciled with project goals during the technical review process as described in Sections 3.0 through 

10.0. 

For the 304(m) annual review process, ERG will compare findings to known values to verify results, 

including overall industry trends, pollutant loadings, and geographic scope. For example, ERG will use 

results from previous years’ findings, including published screening-level databases, the TRIReleases and 

DMRLoads databases and the most recent Preliminary and/or Final ELG Plans described in Section 3.0 of 

this PQAPP. Additionally, ERG will report how well data collected and analyzed for the 304(m) project 

answer the key questions identified in Section 2.2 of this PQAPP. 

Completed questionnaires and sampling data are primary data sources that ERG will use to assist EPA in 

answering, at least in part, each of the key questions posed in Section 2.2 as part of industry studies and 

ELG development. Questionnaire responses will provide detailed descriptions of facility characteristics, 

operations, wastewater generation, pollutant presence, and treatment technologies in use. Questionnaire 

data, in combination with other data sources such as sampling and existing data, will be used in a variety 

of calculations and analyses throughout an industry study or ELG rulemaking. ERG will thoroughly 
Revision No. 1 

Date: October 2013 

Section No. 12 

Page 108 of 109 

document all primary data calculations and analyses for the project file in spreadsheets, databases, 

calculations, database programs and queries, and technical memoranda and documents. ERG will use only 

acceptable questionnaire data in calculations and analyses, and any data limitations or assumptions will be 

clearly described to the user. 
Revision No. 1 

Date: October 2013 

References 

Page 109 of 109 

13.0 REFERENCES

Bicknell, Betsy and TJ Finseth. 2006. Eastern Research Group, Inc. Memorandum to Carey Johnston and 

Jan Matuszko, U.S. EPA. “Comments Received Regarding POTW Removals.” September 8, 2006. 

Codding, Ellie and Deb Bartram. 2005. Eastern Research Group, Inc. Memorandum to the 2006 Effluent 

Guidelines Program Plan Docket, “Publicly-Owned Treatment Works (POTW) Percent Removals Used 

for the TRIReleases2002 Database.” August 12, 2005. 

CREM, 2009. Council for Regulatory Environmental Monitoring (CREM). Guidance on the 

Development, Evaluation and application of Environmental Models. EPA/100/K-09/003. March 2009. 

ERG, 2009. ERG. 304m Project File Management Plan. June 2009. 

ERG, 2012. ERG. Environmental Engineering Support for Clean Water Regulations Quality 

Management Plan. June 2012. 

ERG, 2012b. ERG. Security Plan for Handling Confidential Business Information Under the Clean Water 

Act Contract EP-C-12-021.November 9, 2012. 

EPA, 2001a. U.S. Environmental Protection Agency. EPA Requirements for Quality Management Plans 

QA/R-2. Office of Environmental Information. March 22, 2001, reissued May 31, 2006. 

EPA, 2001b. U.S. Environmental Protection Agency. EPA Requirements for Quality Assurance Project 

Plans QA/R-5. EPA/240/B-01/003. Office of Environmental Information. March 2001, reissued May 21, 

2006. 

EPA, 2002. U.S. Environmental Protection Agency. Guidelines for Ensuring and Maximizing the Quality, 

Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency. 

EPA/260R-02-008. October 2002. 

EPA, 2009. U.S. Environmental Protection Agency. Memorandum from Richard Reding, EAD/OST to 

Regional Quality Assurance Managers. Quality Assurance and Quality Control Requirements in Methods 

Not Published by EPA. May 7, 2009. 

EPA, 2012. U.S. Environmental Protection Agency. Technical Users Background Document for the 

Discharge Monitoring Report (DMR) Pollutant Loading Tool. 2012. 
Attachment A 

PUBLICLY-OWNED TREATMENT WORKS (POTW) PERCENT REMOVALS 

USED FOR THE TRIRELEASES2002 DATABASE 

 
Attachment B 

COMMENTS RECEIVED REGARDING POTW REMOVALS 

 
Attachment C 

TELEPHONE CONTACT REPORT 

 
Attachment D 

304(m) PROJECT FILE MANAGEMENT PLAN 

 
Attachment E 

QUESTIONNAIRE MAILOUT CHECKLIST 

 
Attachment F 

DATA REVIEW GUIDELINES 

 
Attachment G 

METHOD SPECIFIC CHECKLIST 

 
Attachment H 

DELIVERABLE REVIEW SIGN-OFF SHEET 

 
Attachment I 

QA AUDIT CHECKLIST