# Environmental Engineering Support for Clean Water Regulations Programmatic Quality Assurance Project Plan (PQAPP)

Prepared for:



## U.S. Environmental Protection Agency Engineering and Analysis Division Office of Water 1200 Pennsylvania Avenue, NW Washington, D.C. 20460

Prepared by:


## Eastern Research Group, Inc. 14555 Avion Parkway Suite 200 Chantilly, VA 20151

October 2013

Contract No. EP-C-12-021Revision No. 1 Date: October 2013 Page i of viii

Environmental Engineering Support for Clean Water Regulations Programmatic Quality Assurance Project Plan (PQAPP) APPROVAL SHEET

Signatures indicate approval of this Programmatic Qu ality Assurance Project Plan and commitment to follow the applicable procedures noted:

Mary Willett - ERG Contract Quality Assurance Manager Date

Deb Bartram - ERG Program Manager Date

Marion Kelly – EAD Quality Assurance Coordinator Date

Meghan Hessenauer - EPA Technical Project Officer (ERG Support Contract) Date

Jan Matuszko - EPA Branch Chief, Engineering and Analytical Support Branch Date

Jan Goodwin - EPA Branch Chief, T echnology and Statistics Branch DateRevision No. 1 Date: October 2013 Page ii of viii

History of Environmental Engineering Support for Clean Water Regulations Programmatic Quality Assurance Project Plan (PQAPP)

Revision 0 Delivery Date May 15, 2013 Work Assignment 0-40 Scope Covers work performed for EAD under Contract EP-C12-021including primary and existing data collection, generation, evaluation, analysis, and use.
Comments Received EPA Comments on First Draft; received 12/6/2013 EPA Comments on Second Draft: received 3/13/2013 EPA Comment on Table 10-1: received 5/14/2013
Comments Addressed ERG Second Draft PQAPP delivered 1/10/2013 ERG Third Draft PQAPP (without Section 6) delivered 3/22/2013 ERG Final PQAPP (without Section 6) delivered via FedEx 5/15/2013 Approval ERG Approvals 5/15/2013 EPA Approvals 5/29/2013

Revision 1 Delivery Date October 28, 2013 Work Assignment 0-40 Scope Covers work performed for EAD under Contract EP-C12-021including primary and existing data collection, generation, evaluation, analysis, and use.
Comments Received EPA request to incorporate final version of Section 6
Comments Addressed ERG PQAPP delivered 10/28/2013 Approval ERG Approvals 10/28/2013Revision No. 1 Date: October 2013 Page iii of viii

TABLE OF CONTENTS

Page



### 1.0 I NTRODUCTION 

### .................................................................................................................... 1

2.0 P ROJECT 

### M ANAGEMENT 

### E LEMENTS 

### .................................................................................... 3 2.1 PQAPP Distribution ............................................................................................................ 3 2.2 Problem Definition/Background ......................................................................................... 4 2.3 Project/Task D escription ..................................................................................................... 6 2.3.1 EAD’s Annual 304(m) Review Process ................................................................ 7 2.3.2 Preliminary and Detailed Industry Studies .......................................................... 10 2.3.3 Effluent Limitations Guidelines and Standards Development ............................. 10 2.4 Project Organization ......................................................................................................... 12 2.5 Overall Quality Criteria and Objectives ........................................................................... 20 2.6 Special Training/Certification ........................................................................................... 21 2.7 Quality Assurance Program Documents and Records ...................................................... 22 2.8 Reports to Management .................................................................................................... 22



### 3.0 C OLLECTION OF 

### E XISTING 

### D ATA FOR 

### U SE IN 

### C ONDUCTING 

### EAD’ S 

### A NNUAL 

### 304( M 

### ) I NDUSTRY 

### R EVIEW 

### P ROCESS 

### .............................................................................................. 24 3.1 Data Gathering ................................................................................................................ .. 26 3.1.1 Data Sources ........................................................................................................ 26 3.1.1.1 Data Sources for the TRA and Odd-Numbered Year Analysis ........... 26 3.1.1.2 Additional Data Sources ...................................................................... 27 3.1.2 Criteria for Selecting Data Sour ces and Resolving Data Gaps ............................ 29 3.1.3 Documentation and Records ................................................................................ 31 3.2 Data Management ............................................................................................................. 3 1 3.3 Data Quality Assessment .................................................................................................. 32 3.3.1 DMR Pollutant Loading Tool Data Qu ality Review and Verification ................ 32 3.3.2 Other Existing Data Quality Review and Verification ........................................ 35 3.3.3 Achieving the Contract’s Overall Quality Objectives ......................................... 35



### 4.0 C OLLECTION OF 

### E XISTING 

### D ATA FOR 

### U SE IN 

### EAD’ S 

### I NDUSTRY 

### S TUDIES AND 

### ELG D EVELOPMENT 

### ................................................................................................................... 38 4.1 Data Gathering ................................................................................................................ .. 38 4.1.1 Data Needs and Potential Data Sources ............................................................... 38 4.1.2 Literature Searches .............................................................................................. 41 4.1.3 Criteria for Selecting Data Sour ces and Resolving Data Gaps ............................ 42 4.1.4 Data Value Selection Approach ........................................................................... 43 4.1.5 Data Gathering Documentation and Records ...................................................... 44 4.2 Data Management ............................................................................................................. 4 4 4.3 Existing Data Quality Assessment .................................................................................... 45 4.3.1 Quality Review of Existing Data Used in Characterizing Industry Point Sources, Wastestreams, and Treatment Technologies ........................................................ 45 4.3.2 Quality Review for Existing Data Used in Environmental Assessment Analysis49 4.3.2.1 Qualitative Analyses ............................................................................ 49 4.3.2.2 Quantitative Assessments .................................................................... 50 4.3.3 Data Quality Verification and Reporting ............................................................. 51 4.3.4 Achieving the Contract’s Overall Quality Objectives ......................................... 51Revision No. 1 Date: October 2013 Page iv of viii

TABLE OF CONTENTS (Continued)

Page



### 5.0 G ENERATION OF 

### P RIMARY 

### D ATA THROUGH 

### Q UESTIONNAIRES AND 

### S URVEYS 

### ................... 53 5.1 Data Gathering ................................................................................................................ .. 53 5.1.1 Questionnaire Data Collection Process Design ................................................... 53 5.1.2 Questionnaire Data Collection Methods .............................................................. 54 5.1.3 Questionnaire Data Handling and Custody .......................................................... 56 5.1.4 Non-Direct Measurements ................................................................................... 56 5.1.5 Site Visit Primary Data Collection ...................................................................... 56 5.2 Questionnaire Data Management ...................................................................................... 57 5.2.1 Transfer of Electronic Questionnaire R esponses into a Questionnaire Database 57 5.2.2 Transfer of Paper Questionnaire Respon ses into a Questionnaire Database ....... 59 5.2.3 Questionnaire Data Documentation ..................................................................... 59 5.2.4 Management of CBI Data in a Questionnaire ...................................................... 60 5.3 Quality A ssessment ........................................................................................................... 60 5.3.1 Questionnaire Quality Control ............................................................................. 60 5.3.2 Questionnaire Data Acceptance Criteria .............................................................. 63 5.3.3 Site Visit Primary Data Revi ew and Acceptance Criteria ................................... 64 5.3.4 Achieving the Contract’s Overall Quality Objectives ......................................... 65



### 6.0 G ENERATION OF 

### P RIMARY 

### D ATA 

### T HROUGH 

### S AMPLING AND 

### A NALYSIS 

### ............................ 66 6.1 Field Sampling ................................................................................................................ .. 67 6.1.1 Sampling Process Design ..................................................................................... 67 6.1.2 Sampling Methods ............................................................................................... 68 6.1.2.1 Inspection/Acceptance Criteria for Supplies, Consumables, and Sampling Equipment ........................................................................... 69 6.1.2.2 Sampling Equipment Calibration, Ma intenance, and Decontamination ............................................................................................................. 70 6.1.2.3 Methods for Minimizing Sample Contamination ................................ 71 6.1.2.4 Methods for Identifying Upset Conditions Requiring Suspension of Sample Collection ............................................................................... 71 6.1.3 Sample Handling and Custody ............................................................................ 71 6.1.4 Field Data Management ....................................................................................... 73 6.1.4.1 Direct Measurements ........................................................................... 73 6.1.4.2 Non-Direct Measurements ................................................................... 73 6.1.5 Field Quality Control Measures and Goals .......................................................... 74 6.1.5.1 Trip Blank ............................................................................................ 75 6.1.5.2 Bottle Blank ......................................................................................... 75 6.1.5.3 Field Blank .......................................................................................... 76 6.1.5.4 Equipment Blank ................................................................................. 76 6.1.5.5 Field Duplicates ................................................................................... 76 6.1.6 Sampling Program Data Quality Criteria and Goals ............................................ 77 6.1.6.1 Completeness ....................................................................................... 79 6.1.6.2 Sensitivity ............................................................................................ 79 6.1.6.3 Precision .............................................................................................. 79 6.1.6.4 Representativeness .............................................................................. 80 6.1.6.5 Sample Contamination ........................................................................ 80 6.1.6.6 Comparability ...................................................................................... 80Revision No. 1 Date: October 2013 Page v of viii

TABLE OF CONTENTS (Continued)

Page

6.2 Requirements for Laboratories Participating in EAD Sampling Programs ...................... 80 6.3 Laboratory Analysis .......................................................................................................... 81 6.3.1 Laboratory Sample Handling and Custody .......................................................... 82 6.3.2 Analytical Methods .............................................................................................. 82 6.3.3 Instrument/Equipment Calibr ation and Frequency .............................................. 83 6.3.4 Analytical Quality Control ................................................................................... 83 6.3.4.1 Method Detection Limits (MDLs) ....................................................... 83 6.3.4.2 Ongoing Precision and Recove ry (OPR) Standard .............................. 84 6.3.4.3 Calibration Check Standard ................................................................. 84 6.3.4.4 Laboratory Blanks ............................................................................... 84 6.3.4.5 Matrix Spike/Matrix Spike Duplicate/Laboratory Duplicate .............. 85 6.3.5 Preparation and Analytical Batches ..................................................................... 86 6.3.6 Internal Standards, Labele d Compounds, Surrogates .......................................... 86 6.3.7 Non-Standard EPA Wastewater Methods ............................................................ 86 6.3.8 Analytical Data Management .............................................................................. 87 6.3.8.1 Analytical Data Review ....................................................................... 87 6.3.8.2 Analytical Data Reduction .................................................................. 89 6.3.9 Analytical Data Reporting ................................................................................... 90 6.4 Achieving the Contract’s Overall Quality Objectives ...................................................... 91



### 7.0 G ENERATION 

### , E VALUATION 

### , AND 

### / OR 

### A NALYSIS OF 

### D ATA 

### U SING 

### S PREADSHEETS 

### ............ 92 7.1 Developing Spreadsheets .................................................................................................. 92 7.2 Quality Control and Data Quality Goal ............................................................................ 92 7.3 Documenting Spreadsheet Quality Control ...................................................................... 93 7.4 Identification and Control of CBI in Spreadsheets ........................................................... 93 7.5 Quality Assurance ............................................................................................................. 93



### 8.0 G ENERATION 

### , E VALUATION 

### , AND 

### / OR 

### A NALYSIS OF 

### D ATA 

### U SING 

### D ATABASES 

### ................. 94 8.1 Developing Databases ....................................................................................................... 94 8.2 Developing Databa se Queries ........................................................................................... 94 8.3 Quality Control and Data Quality Goal ............................................................................ 95 8.3.1 QC for Database Data Entry ................................................................................ 95 8.3.2 QC for Database Quer ies and Outputs ................................................................. 96 8.4 Documenting Database Quality Control ........................................................................... 96 8.5 Identification and Control of CBI in Databases ................................................................ 96 8.6 Quality Assurance ............................................................................................................. 96



### 9.0 G ENERATION 

### , E VALUATION 

### , AND 

### / OR 

### A NALYSIS OF 

### D ATA 

### U SING 

### M ODELS 

### ....................... 97 9.1 ERG Generated Computational Models ........................................................................... 97 9.2 Existing Models ............................................................................................................... . 98 9.2.1 Review of Data Quality in Existing Models ........................................................ 99 9.2.2 Review of Model Methodology ........................................................................... 99 9.3 Additional Resources ...................................................................................................... 100



### 10.0 D ELIVERABLES 

### R EVIEW 

### ................................................................................................... 101 10.1 Levels of Review ............................................................................................................ 1 01Revision No. 1 Date: October 2013 Page vi of viii

TABLE OF CONTENTS (Continued)

Page

10.2 Types of Review ............................................................................................................. 1 03 10.2.1 Technical Review .............................................................................................. 103 10.2.2 Copyediting Review .......................................................................................... 103 10.2.3 Technical Edit Review ....................................................................................... 104 10.3 Quality Assurance for De liverable Review .................................................................... 104



### 11.0 ERG’ S 

### Q UALITY 

### P ROGRAM 

### A SSESSMENTS AND 

### R ESPONSE 

### A CTIONS 

### .............................. 105 11.1 ERG Quality Program Assessments ............................................................................... 105 11.2 Response Actions ............................................................................................................ 1 06



### 12.0 A SSESSING 

### D ATA 

### U SABILITY AND 

### R ECONCILIATION WITH 

### U SER 

### R EQUIREMENTS 

### ........... 107 12.1 Assessment of Data Usability ......................................................................................... 107 12.2 Reconciliation with User Requirements ......................................................................... 107



### 13.0 R EFERENCES 

### .................................................................................................................... 109

Attachment A: PUBLICLY-OWNED TREA TMENT WORKS (POTW) PERCENT REMOVALS USED FOR THE TRIRELEASES2002 DATABASE

Attachment B: COMMENTS RECEIVED REGARDING POTW REMOVALS

Attachment C: TELEPHONE CONTACT REPORT Attachment D: 304(m) PROJECT FILE MANAGEMENT PLAN Attachment E: QUESTIONNAIRE MAILOUT CHECKLIST

Attachment F: DATA REVIEW GUIDELINES

Attachment G: METHOD SPECIFIC CHECKLIST

Attachment H: DELIVERABLE REVIEW SIGN-OFF SHEET Attachment I: QA AUDIT CHECKLISTRevision No. 1 Date: October 2013 Page vii of viii

LIST OF TABLES

Page



### 2-1 PQAPP Distribution List.................................................................................................... ..3

2-2 Work Activities and Typical Key Questions by Program ...................................................4

2-3 Contract EP-C-12-021 EPA Proj ect Staff Responsibilities ...............................................15

2-4 Contract EP-C-12-021 ERG Proj ect Staff Responsibilities ...............................................16

3-1 Additional Data Sources Identified for the Even-Numbered Annual Reviews .................27

3-2 304(m) Annual Review Data Sources Acceptance Criteria ...............................................36

4-1 Existing Data Purposes and Data Elements Gathered .......................................................39

4-2 Existing Data Acceptance Criteria for Industry Studies and ELG Development ..............47

5-1 Industry Questionnaire Da ta Acceptance Criteria .............................................................64

6-1 Inspection and Acceptance Criteria fo r Supplies, Consumables, and Sampling Equipment ..................................................................................................................... .....69

6-2 Field Quality Control Samples...........................................................................................74

6-3 Field Sampling and Analytical Data Quality Review Criteria ...........................................77

6-4 Data Quality Qualifiers for Sampling Program .................................................................88

6-5 Typical Analytical Database Structure ..............................................................................90

10-1 Levels of Review ......................................................................................................... ....102Revision No. 1 Date: October 2013 Page viii of viii

LIST OF FIGURES

Page



### 2-1 Integration of the 304(m) Annual Revi ew Process with EAD Studies and ELG Development ..................................................................................................................6

2-2 304(m) Annual Review Process .....................................................................................9

2-3 Effluent Guidelines Development Process ..................................................................11

2-4 ERG Program Management Stru cture for Contract EP-C-12-021 ..............................14Revision No. 1 Date: October 2013 Section No. 1 Page 1 of 109

A Supplemental QAPP provides project/work assignment-specifics and/or procedures that are not covered by the PQAPP or that have been added or modified into the work. 

### 1.0 I NTRODUCTION

The scope of this Programmatic Quality Assurance Project Plan (PQAPP) is Eastern Research Group, Inc.’s (ERG’s) Environmental Engineering Support fo r Clean Water Regulations Contract (Engineering Contract) EP-C-12-021 and encompasses the activities included in the contract Performance Work Statement (PWS). This PQAPP covers the collection, generation, evaluation, analysis, and use of environmental data. ERG intends this PQAPP to be used as is or in concert with a Supplemental QAPP (SQAPP) to explain ERG’s quality assurance and quality control (QA/QC) procedures for conducting work under this contract. This PQAPP covers the following types of programs that ERG supports for the U.S. Environmental Protection Agency (EPA) Engineering and Analysis Division (EAD):

 304(m) annual review process;

 Preliminary and detailed studies of specific industries and point source categories; and

 Development and implementati on of effluent limitation guide lines and standards (ELGs).

In support of these programs, ERG conducts the foll owing types of activities that will require QA/QC:

 Collection and analysis of existing data for use in conducting EAD’s annual 304(m) industry review process;

 Collection and analysis of existing data for u se in EAD’s industry studies and ELG development;

 Collection/generation and evaluation of prim ary data through questionnaires or surveys (including written and/or telephone surveys th at may or may not require an Information Collection Request [ICR] and primary data collected during site visits);

 Collection/generation and evaluation of prim ary data through sampling and analysis;

 Generation, evaluation, and analys is of data using spreadsheets;

 Generation, evaluation, and anal ysis of data using databases;

 Generation, evaluation, and analysis of data from ERG designed cost and loading models, existing Agency models, and ERG-desi gned or off-the-shelf models; and

 Preparation of draft and final deliverables.

In preparing work assignments to be issued by EAD under ERG’s Contract EP-C-12-021, the EPA Work Assignment Manager (WAM) and EAD Quality Assura nce Coordinator will review the work assignment activities against the activities covered by this PQA PP and any existing project-specific QAPPs. The EPA WAM will determine what work assignment activities w ill be covered under this PQAPP, what activities will require a SQAPP to be developed, and whethe r a separate project-specific QAPP will be required. These requirements will be specified in the work assignment issued by EAD.

Offices and divisions other than EAD that issue work under ERG’s Contract EP-C-12-021 should evaluate their QA/QC needs and determine if this P QAPP fulfills part or all of those needs. The officesRevision No. 1 Date: October 2013 Section No. 1 Page 2 of 109

and divisions will specify in each work assignment wh ether the work is covered under this PQAPP, an SQAPP is required, or a project-specific QAPP is required.

This document is organized as follows:

 Section 2.0 presents Project Management elements that are generic to all ERG EAD projects and work assignments conducted under this contract;

 Sections 3.0 – 10.0 present activity-specific data gathering, data ma nagement, data quality assessment, and data verifica tion and validation elements;

 Section 11.0 discusses ERG’s internal quality pr ogram assessments and response actions that apply to EAD projects/work assignments conducted under th is contract; and

 Section 12.0 discusses ERG’s approach for assessing data usability and rec onciliation with user requirements for EAD projects/work assignments conducted under this contract.

EAD’s effluent guidelines work is subject to unplann ed deviations as projects/work assignments progress. This PQAPP acknowledges the likelihood of devia tions over time. ERG will update this document, SQAPPs, and project-specific QAPPs to reflect chan ges as appropriate. ERG staff is responsible for implementing this PQAPP’s QA/QC procedures under th is contract. EPA’s WAMs are responsible for specifying the QA/QC documentation required for any pub lic record or docket created under this contract.Revision No. 1 Date: October 2013 Section No. 2 Page 3 of 109



### 2.0 P ROJECT 

### M ANAGEMENT 

### E LEMENTS



### 2.1 PQAPP Distribution

The PQAPP distribution list will include the staff identified in Table 2-1:

Table 2-1. PQAPP Distribution List

Name Title Contact Information Mailing Address
U.S. Environmental Protection Agency , Engineering and Analysis Division

Meghan Hessenauer Technical Project Officer (202) 566-1040 Hessenauer.Meghan@epa.gov

U.S. Environmental Protection Agency Office of Science and Technology 1200 Pennsylvania Avenue, NW (4303T) Washington, DC 20460 (202) 566-1053 (fax) Marion Kelly EAD Quality Assurance Coordinator (202) 566-1045 Kelly.Marion@epa.gov

Jan Matuszko Branch Chief, Engineering and Analytical Support Branch (202) 566-1035 Matuszko.Jan@epa.gov

Jan Goodwin Branch Chief, Technology and Statistics Branch (202) 566-1060 Goodwin.Janet@epa.gov

EAD Work Assignment Manager Specified in the EPA Work Assignment

Eastern Research Group, Inc.

Deborah Bartram ERG Program Manager (703) 633-1669 Deborah.Bartram@erg.com

Eastern Research Group, Inc. 14555 Avion Parkway Suite 200 Chantilly, VA 20151-1102 (703) 263-7280 (fax) Mary Willett ERG Contract Quality Assurance Manager (703) 633-1615 Mary.Willett@erg.com

ERG Work Assignment Manager Specified in ERG Work Plan

ERG Work Assignment Project Team 1 Specified in ERG Work Plan

1 The ERG project team will include the Project QA Co ordinator and may include subcontractor staff. If subcontractor staff are used to support a work assignmen t, the ERG WAM will provide the subcontractor manager, QA Manager (if applicable), and techni cal staff with applicable portions of this PQAPP and will review appropriate QA/QC procedures at th e start of the work.Revision No. 1 Date: October 2013 Section No. 2 Page 4 of 109



### 2.2 Problem Definition/Background

Each EAD work assignment, work assignment amendmen t, and/or technical directive issued under the contract will include details about the background of the technical work and will define the activities required by the project. Table 2-2 presents the typical work activities that ERG will support and the key questions that define the problems addressed in each EAD program that will be conducted under the

contract.

Table 2-2. Work Activities and Typical Key Questions by Program

Program Work Activities Typical Key Questions to be Answered by the Project Work Statement
304(m) Annual Review ERG’s analysis of existing data sources will provide technical support for EPA preparation of Effluent Guidelines Program Plans required by CWA Section 304(m). ERG will provide technical support for the following activities:

 Annual review of effluent guidelines and pretreatment standards that EPA has promulgated under the Clean Water Act (CWA);

 Identification of cat egories of sources discharging toxic or nonconventional pollutants for which guidelines or pretreatment standards have not previously been published;

 Establishing a schedule for further investigation and possible development or revision of effluent guidelines and pretreatment standards;

 Preparation of technical documents;

 Preparation of regulatory rulemaking records and public dockets; and

 Evaluation of public comments. To prepare the Final Effluent Guidelines Program Plans, EAD will seek to answer, among others, the following questions:

 What industrial wastewater discharges pose the greatest potential threat to human health or the environment?

 What industrial wastewater discharges should be studied further?

 What industrial wastewater discharges warrant possible ELG development or revision?

EAD Industry Studies and/or ELG Development ERG’s collection of primary and existing data will support EPA’s technical understanding of industry profiles, industry processes that generate wastewaters subject to the CWA, pollutant discharges and their impacts, pollutant controls, and the costs and benefits of pollutant controls. Primary and existing data are also typically used to support environmental data evaluation and analysis.

ERG will provide technical support for the following types of activities:

 Engineering site visits; In collecting primary and existing data as part of an industry study or effluent guideline development project, EAD seeks to answer the following questions:

 What facilities are included in a specified industry category? Where are they located? What is their size?

 What are the primary sources of wastewater within an industry category?

 What are the characteristics of the raw wastewaters (i.e., pollutants, concentrations, flow rate/volume)?

 What pollutants are discharged in the final effluent of an industry category operation?

 What pollutant discharges requireRevision No. 1 Date: October 2013 Section No. 2 Page 5 of 109

Table 2-2. Work Activities and Typical Key Questions by Program

Program Work Activities Typical Key Questions to be Answered by the Project Work Statement

 Field sampling activities;

 Questionnaire development and implementation;

 Technical literature reviews;

 Industry profile development;

 Technology options development;

 Technology performance assessments;

 Engineering cost estimation;

 Evaluation of regulatory alternatives;

 Preparation of technical documents;

 Preparation of regulatory rulemaking records and public dockets;

 Evaluation of public comments; and

 Rulemaking implementation support. additional control? What are the pollutants of concern?

 What are the concentrations and loadings of pollutants currently being discharged (i.e., “baseline” concentration)?

 What are the environmental impacts resulting from discharge of pollutants of concern? What is the toxicity associated with the pollutants of concern?

 What are the best available technologies for pollution prevention and wastewater treatment that reduce discharges of pollutants of concern and to what levels?

 How will industry discharges change if plants control pollutants using these technologies/practices?

 What are the capital and operating costs of the best available technologies/practices?

 What are the environmental impacts associated with baseline conditions? What are the environmental impacts associated with various regulatory options?

Figure 2-1 presents the integration of the 304(m) a nnual review process with EAD’s selection of industry categories for studies and ELG development. The 30 4(m) annual review informs EAD’s decisions on which industry categories to evaluate further a nd which categories will be addressed through an ELG. Based on the decisions made by EAD as part of th e 304(m) annual review, ERG may be directed to support a preliminary or detailed study and/or suppor t an ELG development program. ERG’s support to a study and/or ELG development includes collection of existing and primary data, evaluation and analysis of those data to assess wastewater discharges, pollutant s, discharge loads, and applicable treatment and/or pollution prevention technologies to reduce pollutant lo ads, documentation of analysis through draft and final deliverables, preparation of rulemaking record s, and review and evaluation of public comments. ERG also develops calculation tools and models to assess industry-specific costs and load removals that result from application of regulatory options and en vironmental impacts associated with baseline and post-regulation discharges.

ERG’s 304(m), preliminary or detailed study, and EL G support is conducted using an interactive process with EPA. During data collection, ERG will assess inform ation collected and/or identified, will discuss its quality and usefulness with the EPA WAM and, based on those discussions, may conduct additional data collection or may proceed with data analysis sup port. During ERG’s data analysis support activities, analysis results are checked as described in this P QAPP and reviewed with the EPA WAM. Based on this collaborative discussion of analysis and evaluation, EPA will direct ERG to collect additional data, perform additional analysis, conduct additional evalua tion, and/or prepare documentation on the analysisRevision No. 1 Date: October 2013 Section No. 2 Page 6 of 109

results. While ERG may recommend use of specific data or approaches, the EPA WAM will be responsible for deciding data utility.

Figure 2-1. Integration of the 304(m) Annual Review Process with EAD Studies and ELG Development



### 2.3 Project/Task Description

This section presents general program-level descriptions of activities related to the 304(m) annual review process, industry studies, and ELG development projects. EPA issues individual work assignments to ERG for specific projects with pr oject-specific task descriptions. 2 EPA’s work assignments will specify which PQAPP sections apply to the work to be co nducted under the work assignment. As necessary, ERG will supplement this PQAPP with project-specific S QAPPs, study plans, and sampling and analysis plans to identify the type(s) of data that will be gathered or generated, and where and how the data will be obtained. The programs supported under the EAD contr act are described below in terms of program purpose and types of activities conducted. For each of these programs, ERG will generate project-related documentation and deliverables as part of the pr oject file. As needed, ERG uses project file documentation and EPA-directed deliverables to popu late a public record. Also, for each of these programs ERG may use spreadsheets, da tabases, and/or modeling to conduct the work. Sections 7.0, 8.0, and 9.0 of this PQAPP present details on ERG’ s QA/QC activities while using Microsoft Excel® spreadsheet, Microsoft Access® database, and modeling to ols, respectively, as part of our data evaluation and analysis efforts. Section 10.0 presents de tails on ERG’s QA/QC activities associated with development and finalization of written project deliverables.

2 Projects supporting the development of effluent guidelines may span multiple work assignments. In addition, some work assignments, such as those supporting EPA’s annual review and detailed industry studies, may comprise multiple projects.Revision No. 1 Date: October 2013 Section No. 2 Page 7 of 109

EPA Contract EP-C-12-021 has a contract period of performance of September 26, 2012, through September 25, 2017. All work will be performed dur ing this period of performance and in accordance with the timeframes specified in indivi dual work assignments issued by EPA.



### 2.3.1 EAD’s Annual 304(m) Review Process

The 1972 CWA directs EPA to develop national regu lations placing limits on the pollutants that are discharged by categories of industry to rivers and streams (termed “effluent guidelines”) or to sewage treatment plants 3 (termed “pretreatment standards”). The Act also directs EPA to develop national regulations for new industrial facilities (termed “n ew source performance standards”). An additional critical component of the Act is that it requires EPA to periodically study and review existing effluent guidelines, pretreatment standards, and new source pe rformance standards and to consider the need to develop regulations for industries not covered by a national regulation. These planning requirements are found in several sections of the CWA. Section 304(m) provides for an effluent guideline plan that contains three basic elements to be published on February 4, 1987, and biennially thereafter.

First, EPA must establish a schedule for the annual review of existing effluent guidelines promulgated under Section 304(b) (i.e., limitations for existing di rect dischargers) and for annual revision of the guidelines if appropriate (see Section 304(m)(1)(A)). Second, EPA must identify categories of sources that directly discharge toxic or non-conventional pollutants for which EPA has not published effluent limitations guidelines or new source performance sta ndards (see Section 304(m)(1)(B)). Third, EPA must establish a schedule for the establishment of national regulations for any categories identified in the second step, with a promulgated regulation three years after identification in a national plan (see Section 304(m)(1)(C)).

For indirect dischargers, Section 304(g) requires EPA to review at least annually and, if appropriate, revise the pretreatment standards EPA has promulga ted under CWA Section 307. In addition, Section 307(b) provides that EPA must promulgate pretreatment standards for categories of sources not subject to existing pretreatment standards if there is pass-th rough or interference at POTWs. As a good government practice, EPA publishes the findings of its annual review s of direct and indirect dischargers together in one document, the “Effluent Guidelines Program Plan” (ELG Plan). Typically, EPA publishes a preliminary Plan in odd-numbered years and publish es a final Plan in even-numbered years after public review on the preliminary Plan.

To support EPA’s annual review process, ERG co llects existing data from EPA’s National Pollutant Discharge Elimination System (NPDES), the Toxics Re lease Inventory (TRI) program, and other existing data sources as described in Section 3.1.1 to help address th e questions identified in Table 2-2. ERG and EPA’s evaluation of these data support assessment of which facilities and industry categories are discharging the largest amounts of toxic weighted pollutants. For those industries that discharge significant toxic pollutants that are not already su bject to effluent limitations and guidelines, EPA assesses whether additional study of the industry is warranted. ERG also evaluates whether there have been significant changes within industry categories subj ect to existing ELGs that might require revision of the regulations.

3 Also referred to as publicly owned treatment works (POTWs).Revision No. 1 Date: October 2013 Section No. 2 Page 8 of 109

Figure 2-2 presents the basic steps of EPA’s annual review process for odd- and even-numbered years. ERG supports EPA by supplementing the industry categories of interest identified through the evaluation of NPDES and TRI data with other sources of existi ng data to identify new industry categories that need to be assessed.

Based on the annual review process, EPA will prioriti ze for further review (i.e., a detailed study or preliminary category review) those industrial cate gories whose pollutant discharges potentially pose the greatest hazards to human health or the environmen t because of their toxicity (i.e., categories that collectively discharge over 95 percent of the total to xic weighted pounds). EPA also considers efficiency and implementation issues raised by stakeholders in identifying candidates for further review. By using this layered approach, the Agency concentrates its resources on those point source categories with the highest estimates of toxic-weighted pollutant disch arges (based on best available data), while assigning lower priority to categories that the Agency believes are not good candidates for ELG revision.

Table 2-2 presents a listing of the work activities ERG performs in support of the 304(m) annual review process. Section 3.0 presents details on the existing data sources evaluated for the 304(m) annual review process and the QA/QC activities conducted by ERG when supporting that program.Revision No. 1 Date: October 2013 Section No. 2 Page 9 of 109

Figure 2-2. 304(m) Annual Review ProcessRevision No. 1 Date: October 2013 Section No. 2 Page 10 of 109



### 2.3.2 Preliminary and Detailed Industry Studies

For those industrial point source categories identif ied through the 304(m) annual review process for additional study, EAD initiates a preliminary or detail ed study to collect additional data. EPA uses the data from these studies to assess whethe r the Agency should initiate an ELG rulemaking activity to either establish new or revise existing regulations on wastew ater discharges from that category. ERG’s data collection and analysis process for an industry st udy is similar to that conducted under a full ELG development project; however, the industry study typi cally entails a more limited data collection and analysis effort. To support EPA’s industry studies, ERG collects publicly avai lable information on the universe of facilities in the point source category, wastewaters generated by the industry’s processes, existing characteristics of those wastewaters (e.g., pol lutants, concentrations, flows), and preliminary information on the types of treatment and pollu tion prevention processes used by the industry (or applicable to the wastewaters generated). Using the information ERG collects, EAD will assess and determine whether development of a full ELG is warranted.

Table 2-2 presents a listing of the work activities ERG performs in support of industry studies. Section 4.0 presents details on the existing data sources used to support preliminary and detailed industry studies and the QA/QC activities conducted by ERG when performing these studies.



### 2.3.3 Effluent Limitations Guidelin es and Standards Development

For those industrial point source categories identif ied for ELG development, EAD conducts a series of data collection, evaluation, and analysis steps to identify technology-based regulatory options and calculate associated regulatory limits and standards to control pollutant discharges from the point source. Figure 2-3 presents the basic steps of the ELG pro cess. ERG’s support to EAD in this process includes collection of existing and primary data, evaluation and analysis of these data, and presentation of the analyses in draft and final deliverables that docum ent the data and EAD’s decision making process using the data. ERG supports most of the steps outlined in Figure 2-3 with the exception of designating industries for regulation (Step 1 of Figure 2-3) whic h is performed by EPA, analysis of economic impacts and environmental benefits (Step 7, Step 12, Step 13c , and Step 13(d) of Figure 2-3) which is conducted under a separate contract, calculation of effluent lim its (Step 15 of Figure 2-3) which is done by EPA’s statistical group, and selection and publication of the preferred option (Step 16 of Figure 2-3) which is performed by EPA. ERG supports devel opment of industry costs for various regulatory options which are then used as part of the economic impact analysis a nd development of loads for various regulatory options which are then used as part of the environmental be nefits analysis. ERG may also support environmental analysis and assessments of selected point source categories under study. ERG supports proposal and final regulatory efforts by providing regulatory record materials generated as part of the rule to EPA’s Federal Document Management System.

Table 2-2 presents a listing of the work activities ER G performs in support of EL G development. Sections 4.0, 5.0, and 6.0 of this PQAPP present details on existing and primary data collection activities performed by ERG to support the ELG process and the QA/QC activities conducted by ERG.Revision No. 1 Date: October 2013 Section No. 2 Page 11 of 109

Figure 2-3. Effluent Guidelines Development ProcessRevision No. 1 Date: October 2013 Section No. 2 Page 12 of 109



### 2.4 Project Organization

ERG’s program management structure for Contract EP-C-12-021 is shown in Figure 2-4. Each ERG project under this contract, issued to ERG through in dividual work assignments, will be directed by the EPA WAM designated in the work assignment. ERG’ s Engineering Contract Program Manager, Deb Bartram will assign an ERG WAM who will coordinate the work assignment team at ERG. Deb Bartram will provide technical oversight of all work complete d on the contract and will work in coordination with the EPA Project Officer, Meghan Hessenauer and QA staff. As ERG’s Contract QA Manager, Mary Willett will work with Deb Bartram to implement the quality requirements that EPA has specified for this contract. Ms. Willett will also interact with the E AD Quality Assurance Coordinator, Marion Kelly who will work with each of the EPA WAM’s in determin ing the PQAPP quality requir ements that apply to each individual work assignment. Most work assignm ents/projects under the contract will include an ERG Project QA Coordinator who will work with the pr oject team and the project technical reviewers to ensure that ERG’s QA/QC systems are used. The ERG WAM’s will have overall responsibility to ensure that the project team adheres to the requirements of the PQAPP (or other supplemental and/or projectspecific QAPPs) and they will work with the ERG Project QA Coordinator to support and assess compliance. In addition, ERG has a corporate quality staff made up of a Corporate QA Manager and Local QA Coordinators in each of our main offices. These quality staff act as a resource to all ERG staff in understanding ERG’s quality system and support pr ojects in setting up quality procedures and tools. Work plans developed for each work assignment under the contract will name those ERG individuals fulfilling the roles identified in this figure.

If subcontractors are used on a work assignment, ERG’s WAM will interact with the subcontractor WA Manager and any subcontractor QA Manager. ERG’s WAM will provide the subcontractor staff with a copy of applicable portions of this PQAPP and will ensure that the QA/QC requirements of the work assignment are understood by the subcontractor staff at the start of the work. Any specific quality requirements will be included in the subcontract . The ERG WAM is responsible for ensuring that subcontractor products meet the requirements of th is PQAPP. The ERG Program Manager has ultimate responsibility for ensuring subcontractor compliance with ERG’s PQAPP (or other supplemental and/or project-specific QAPPs). If ERG identifies any qualit y issues while working with a subcontractor, ERG’s Program and Work Assignment Managers and the subcontractor’s Program and Work Assignment Managers will become involved and partic ipate in identifying corrective action.

The responsibilities of EPA project staff by project role are summarized below in Table 2-3. The responsibilities of ERG project staff by project role are summarized below in Table 2-4.

ERG technical staff supporting work under the work assignments of Contract EP-C-12-021 will have various project support roles as specified by ERG’ s work plans. In addition, the ERG WAMs for sampling projects will assign a project staff member to act as Crew Chief for each sampling episode conducted. ERG’s sampling Crew Ch iefs have overall responsibility fo r ERG’s field work and manage and coordinate sampling activities as described in Sec tion 6.0 below. For projects that involve developing and utilizing spreadsheets, the ER G WAM will designate a spreadsheet developer. The spreadsheet developer is responsible for creati ng and using the spreadsheet and ensuring that it meets project needs as described in Section 7.0 below. The ERG WAMs for work assignments involving database development will assign an ERG project staff member(s) to act as database task leader and/or developer and queryRevision No. 1 Date: October 2013 Section No. 2 Page 13 of 109

developer. ERG’s database task leaders are respons ible for coordinating the design and development of databases to meet project needs as described in Section 8.0 below.Revision No. 1Date: October 2013Section No. 2Page 14 of 109Figure 2-4. ERG Program Management Structure for Contract EP-C-12-021Revision No. 1Date: October 2013Section No. 2Page 15 of 109Table 2-3. Contract EP-C-12-021 EPA Project Staff ResponsibilitiesProject RoleSummary of ResponsibilitiesEPA Project Officer The EPA Project Officer is responsible fordefining overall quality objectives and overall direction on thecontract activities. Theduties of the EPA Project Officer include:Participating in meetings with ERG managers, EPA WAMs, the EAD QA Coordinator and other EPA staff concerning datacollection needs and quality assurance matters under the contract.Reviewing major deliverables under the contract, as appropriate.Assisting EPA WAMs, EPA Managers, and staff from other organizations within and outside EPA, in making decisions based onthe data gathered under the contract.Working with the EPA QA Officer and QA Coordinators, ERG QAManager, and ERG managers to identify corrective actionsnecessary to ensure that quality objectives under the contract are met.EPA WAM Each EPA WAM will report to the EPA PO regarding activities assigned to ERG under individual work assignments. The EPAWAMs are responsible for ensuring that the policies described inthe PQAPP are supplemented with work plans, study plans,Sampling and Analysis Plans (SAPs), and other planning documents that detail any additional procedures needed to ensure that datacollection activities performed under their work assignments meetwork assignment needs. EPA WAMs are also responsible for thefollowing:Participating with senior EPA program staff and technical personnel, and as appropriate ERG technical staff, in systematicplanning for projects to be conducted under their work assignment, including the development of project objectives, the associatedmeasures of quality, and acceptance criteria.Identify the resources needed for each project to be performed under their work assignment, including quality system trainingneeds, and requesting funding from the Branch Chief to meet those needs.Identifying and implementing project-specific quality management procedures, which may include data quality assessment,information management, data integration, and data validation.Completing the quality Assurance Review Form for each work assignment involving environmental data operations to indicate thequality system requirements.Ensuring that work assignments, work plans, and contract deliverables include quality system documentations appropriate for theactivity.Reporting any deviations from the QAPPs used under thiscontract to the EAD Quality Assurance Coordinator.Preparing and implementing quality system documentation appropriate for the project.Monitoring contractor performance against the work planand quality system documentation (e.g., this PQAPP).Review the performance of ERG staff.Communicating with and directing the ERG WAM regarding requirements and schedules for tasks to be performed under theRevision No. 1Date: October 2013Section No. 2Page 16 of 109Table 2-3. Contract EP-C-12-021 EPA Project Staff ResponsibilitiesProject RoleSummary of Responsibilitieswork assignment.EAD QA Coordinator The EAD QA Coordinator reports to the EAD Director and is responsible for:Reviewing and approving this PQAPP and any other SQAPPs or project-specific QAPPs developed under the contract.Monitoring whether all QA procedures described in the PQAPP and any other SQAPPs or project-specific QAPPs developedunder the contractare followed.Assisting the EPA WAMs in implementing corrective actionsnecessary to resolve any deviations from the PQAPP.Table 2-4. Contract EP-C-12-021 ERG Project Staff ResponsibilitiesProject RoleSummary of ResponsibilitiesERG Program Manager The ERG Program Manager organizes and directs the technical activities and the contract and functions as the primary liaisonbetween the client, management, and the project team. The duties of the Program Manager include:Communicating with the EPA client on technical matters.Informing project staff of contract requirements and ensuring that the staff follows the requirements.Reviewing contract modifications, work assignments, work plans, and cost estimates.Ensuring that the project is assignedthe appropriate ERG staffing levels subcontractors, and technical expertise.Initiating and reviewing subcontractor authorization to work.Managing and reviewing technical and financial progress reports and invoices.Implementing ERG’sSecurity Plan for Handling Confidential Business Information under the Clean Water Act Contract EP-C12-021.Implementing ERG’s quality management plan (QMP) as specifiedin this PQAPP and any supplemental or project-specificQAPPs prepared to support work assignments issued under the contract.Ensuring that ERG remains free of conflicts of interest.Working with the Project QA Coordinator and the ERGWAMs to resolve any quality problems that arise.Serving as the point of contact with ERG management on all matters relating to the contract.Coordinating the technical components (including personnel, facilities, and equipment) required under the contract.Monitoring the technical activities on projects under the contractto ensure that the technical objectives, budget, and schedule areRevision No. 1Date: October 2013Section No. 2Page 17 of 109Table 2-4. Contract EP-C-12-021 ERG Project Staff ResponsibilitiesProject RoleSummary of Responsibilitiesmet.Reviewing performance of ERG WAMs, subcontractors, and consultants.ERG WAM ERG WAMs coordinate the activities of the individual workassignments performed under the contract. In executing theseduties,ERG WAMs:Are available to the EPA client for action on any problem related to specific work assignments.Responsible for designating ERG staff for preparing any required SQAPPs or project-specific QAPPs.Prepare and implement work plans, test plans,health and safety plans, and associatedproject instructions in response to workassignments and responsible for following the requirements of this PQAPP and/or supplemental QAPP or project-specific QAPPas specified by the EPA WAM.Provide the EPA client with periodic status briefings or reports.Implement ERG’s cost and performance tracking and control system.Complete projects on time, within budget, and in accordancewith the work order or assignment technical and regulatoryobjectives.Maintain and document all project-related records in individual project files (these may include files, calculations, assumptions,and professional engineering judgments).Follow ERG’s PQAPP which complies with ERG’s QMP, to ensurethe technical quality of reports, memoranda, and othercommunications from inception to delivery.Monitor the technical activities of each ERGwork assignment staff member to ensurethat all staff are meeting the highesttechnical standards and adhering to the budget and schedule.Review the performance of ERG staff.Keep the ERG Program Manager informed onall aspects of each task, including expenditures, technical progress, problems, andrecommended solutions.Monitor subcontractor performance and provide performance data to the ERG Program Manager.Ensure compliance with all QC acceptance criteria asspecified in any SQAPP orproject-specific QAPP.Keep the ERG QA staff and the ERG Program Manager advised of any quality problems that arise.Discuss any significant quality deficiencies with the EPA WAM and ensure that actions to correctquality deficiencies are taken,documented, and reported to the ERG Project QA Coordinator.ERG Contract QA The ERG Contract QA Manager is responsible for:Revision No. 1Date: October 2013Section No. 2Page 18 of 109Table 2-4. Contract EP-C-12-021 ERG Project Staff ResponsibilitiesProject RoleSummary of ResponsibilitiesManagerWorking with the ERG Program Manager to develop this PQAPP and ensure that contract-specific QA/QC programs arecommensurate with contract objectives and with ERG’s quality system.Assisting the Program Manager and ERG WAMs in identifying and assigning appropriate Project QA Coordinators and technicalreviewers.Reviewing with Project QA Coordinators the QA/QC procedures developed and executed for each work assignment in accordancewith the PQAPP, SQAPP or individual QAPPs.Conducting independent audits to determine the effectiveness of the Environmental Engineering Support contract QA/QCprogram.Issuing stop work orders for quality considerations.Maintaining records of contract internal QA audits.ERG Project QACoordinatorProject QA Coordinators are responsible for:Implementing this PQAPP and/or supporting preparation of a project-specific QAPP/SQAPP that documents the workassignment-specific policies, organization, objectives, functional activities, and specific QA/QC procedures designed to achievequality goals or requirements as required by the work assignment. This includes reviewing work-assignment-specific QAPPs.Verifying that the requirements of theapproved PQAPP and other ERG QA/QC procedures are communicated to the workassignment staff.Serving as internal consultants to the Program Manager andERG WAMs in defining quality goals or requirements and indeveloping a work-assignment-specific QC system that is responsive to them.In consultation with the project management, verifying that subcontractors’ and consultants’quality-related procedures areadequate and executed.Bringing quality problems to the immediate attention of theERG Program Manager, including a recommendation to stop work,replace staff, or other changes as needed or, if warranted, to the attention of theContract QA Manager for implementation ofcorrective action.Providing an independent assessment of performance through QA audits.Overseeing any external QA audit activities requested by the client if required.Documenting the results of all QA/QC activities in reportsto ERG management and, if required, to the client(s).ERG Technical Reviewer Technical reviewers reviewproject deliverables:To determine if the deliverable objectives havebeen fulfilled in a technically sound manner.Revision No. 1Date: October 2013Section No. 2Page 19 of 109Table 2-4. Contract EP-C-12-021 ERG Project Staff ResponsibilitiesProject RoleSummary of ResponsibilitiesTo ensure that the deliverable is clearly written.To check that calculations are correctand references are fully documented.ERG Sampling CrewChiefThe ERG sampling crew chief has overall responsibility for ERG’s field work on a specific sampling episode and is responsible for:Coordinating development of the Sampling Plan and Health and Safety Plan for a specific sampling episode.Managing and coordinating sampling activities as described in Section 6.0 of this PQAPP.ERG SpreadsheetDeveloperThe ERG spreadsheet developer is responsible for:Creating, naming, and developing the spreadsheet.Ensuring that data in the spreadsheet are imported or enteredcorrectly, clearly identified andlabeled, properly referenced, and thatmathematical functions used for manipulating data are executed accurately as described in Section 7.0.ERG Database TaskLead/DeveloperThe ERG database task lead/developer is responsible for:Coordinating the design and development of databases and/or developing databases to meet project needs.Ensuring that database QC procedures are followed and documented as described in Section 8.0.ERG Project Staff Project staff are responsible for executing the work assignment QA/QC requirements specified in this PQAPP, any project-specificQAPPs/SQAPPs, and other work assignment plans. They are responsible for documenting the results of their QA/QC activities andfor communicating with the Project QA Coordinator and ERG WAM regarding quality issues and concerns.Revision No. 1 Date: October 2013 Section No. 2 Page 20 of 109



### 2.5 Overall Quality Criteria and Objectives

The overall quality needed to support EAD’s 304(m) annual review process, preliminary and detailed studies, and ELG development and implementation project s is discussed below. Sections 3.0 through 6.0 present activity-specific data qualit y criteria and associated data quality specifications and acceptance criteria that apply to ERG’s support of EAD’s programs.

EAD uses EPA’s Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by th e Environmental Protection Agency (EPA, 2002) as the basis for determining the quality of data n eeded to support this contract. These Information Quality Guidelines (IQGs) are designed to ensure the Agency’s complia nce with the Data Quality Act (also known as the Information Quality Act), and address the following goals:

Disseminated information should adhere to a basic sta ndard of quality, including objectivity, utility, and integrity;
The principles of information quality should be in tegrated into each step of EPA’s development of information, including creation, maintenance, and dissemination; and

Administrative mechanisms for correction should be fl exible, appropriate to the nature and timeliness of the disseminated information and incorporated into EPA’s information resources management practices.

As stated in the EPA IQGs, the Ag ency “works every day to ensure information quality, but does not wait until the point of dissemination to consider importa nt quality principles...” The IQGs further define quality based on the following three characteristics:

 Objectivity : (1) The information must be accurate, re liable, and unbiased, and (2) the manner in which the information is presented must be accurate, clear, complete, and unbiased;

 Integrity : The information may not be compromised through corruption or falsification, either by accident, or by unauthorized access or revision, and

 Utility : The information must be usef ul for the intended users.

Within EAD, these IQGs are implemented in accord ance with Office of Water Information Quality Guidelines: Pre-dissemination Review Guidance and Checklists, version 2.3, September 23, 2010. The EPA and OW IQGs distinguish between influential information
, influential scientific risk assessment information , and non-influential information. Based on these requirements, EAD has determined that data will be collected and used on this contract to support influential scientific information . Accordingly, the data collection, manipulation, and analyses performed under this contract must meet the objectivity, integrity, and utility data quality objectives defined above, and must provide “t ransparency regarding (1) the source of the data used, (2) the various assumptions employed, (3) the analytical methods applied, and (4) the statistical procedures employed.”

ERG discusses these overall quality objectives in the c ontext of the data quality procedures of this PQAPP in Sections 3.3.3, 4.3.4, 5.3.4, and 6.4 below.Revision No. 1 Date: October 2013 Section No. 2 Page 21 of 109



### 2.6 Special Training/Certification

ERG has the following in-house training programs that apply to work conducted for EAD under this

contract.

CWA Confidential Business Information (CBI) Security Training/Certification – ERG maintains an approved CWA CBI Security Plan. The plan is review ed annually by any staff that will access or use CWA CBI, and ERG conducts certification for CBI use via a test and authorization form. The ERG Program Manager (Deb Bartram) is responsible for id entifying any additional program staff that will need to access CWA CBI and to inform them and the Docume nt Control Officer (DCO) of the need for staff training. ERG’s DCO or Deputy DCO will issue the plan and test staff as required.

Health and Safety (H&S) Program – ERG has a corporate H&S program that includes a medical monitoring program and a minimum of 24 hours H&S traini ng for staff prior to engineering site visits or sampling participation. ERG conducts an annual 8- hour refresher H&S training for any Chantilly, VA office staff who are expected to conduct site visits and/or sampling episodes. The ERG Program Manager is responsible for identifying any ERG staff anticip ated for site or sampling activities and they are included in ERG’s H&S program. ER G’s Chantilly, VA office H&S coordinator is Steve Strackbein, who plans and coordinates our annual Chantilly, VA o ffice H&S refresher training and additional H&S training as needed.

Sampling Training - ERG has previously conducted and will continue to periodically conduct an inhouse sample collection training session that reviews wastewater sample collection, preservation, packaging, and shipping procedures specific to ERG’ s Office of Water support. As needed, ERG will also conduct training on clean hands/dirty hands sampling techniques specific to the collection of wastewater samples for low-level metals analyses. The ERG Program Manager and appropriate ERG WAMs will identify staff who are anticipated to be deployed fo r sampling and will determine whether and what type of sampling training is needed and at what freque ncy. Generally, ERG will plan to conduct sampling training prior to anticipated sampling programs for EPA’s Office of Water and will ensure that any new staff who has not participat ed in the training attend.

Hazardous Materials Shipping Training – Part of ERG’s annual H&S refresher training includes a review of International Air Transport Association’ s (IATA’s) hazardous materials shipping requirements.

Quality System Training – ERG conducts training on ERG’s corporate quality policy and systems for all technical staff on an annual b asis. ERG’s Corporate QA Manager (Mary Willett) and the Local QA Coordinators are responsible for coor dinating and conducting this training.

Training and certification requirements beyond those lis ted above and identified as necessary as part of a specific project/work assignment will be discussed in ERG’s work plans (or SQAPP) along with our process for conducting/obtaining the necessary tr aining or obtaining the necessary certification.Revision No. 1 Date: October 2013 Section No. 2 Page 22 of 109



### 2.7 Quality Assurance Program Documents and Records

Quality assurance program documentation fo r the EP-C-12-021 contract includes:

 ERG’s customized QMP included in our contract and modified under WA 0-40 to incorporate additional information provided in the QMP Addendum;

 This contract-level PQAPP, which is fully compliant with ERG’s customized QMP;

 Contract-related paperwork, in cluding the project-specific work assignments, work assignment amendments, and technical directives (see Secti on 2.2) and ERG work plans (see Section 2.4);

 Where specified, SQAPPs and/or project-specific QAPPs (see Section 1.0);

 ERG Standard Operating Procedures (SOPs) referenced in ERG’s customized QMP and discussed in Sections 6.0 (sampling procedures), 7.0 (spreadsheet quality control procedures), 8.0 (database quality control procedures), 9.0 (m odel development and use quality control procedures), and 10.0 (deliverable review);

 ERG’s planning documents, including Sampling a nd Analysis Plans (see Section 6.1.1), Study Plans (see Section 2.3.2), and systematic planning process memoranda (see Sections 5.1.1 and 8.1); and

 Project files that discuss and document QA/QC-related activities (see Sections 3.1.3, 3.2, 4.2, 5.1.3, 5.2.3, 6.1.4, and 6.3.8).

Discussions of these project specific documentation and records occur throughout this document. ERG’s QAPPs are “living” documents and subject to modifi cation as the work becomes more fully defined. To maintain version control, each QAPP has a header identifying the revision number of the PQAPP, SQAPP, or other QAPPs and subsequent revisions will include a front table identifying significant changes from the previous version. At the direction of the EAD Quality Assurance Coordinator, ERG will update this PQAPP resulting in a subsequent version. Alternatively, modifications or additions to the QA/QC procedures conducted to support work under th is contract may be covered by a SQAPP or aproject/work assignment-specific QAPP. EPA’s WAMs will direct the need for SQAPPs or projectspecific QAPPs.



### 2.8 Reports to Management

When directed to do so by the EPA WAMs, ERG will develop task-specific quality review documentation to present a summary of data reviews against stated data quality objectives (D QOs) and/or data quality goals. ERG will provide any required data quality documentation to th e EPA WAM and will include it in the project files. Details on quality documentation by t ype of activity are described in subsections 3.3.2, 4.3.3, 5.3.1, 6.1.6, 6.3.8.1, 7.3, and 8.4 belo w. Additionally, ERG will pr ovide each EPA WAM with monthly reports on the status of QA activities. These reports will be prepared by the ERG WAM and incorporated into our monthly technical progress repor ts (TPRs), which will be supplemented to include the following:

 Quality assurance activities performed during the reporting period as part of the implementation of the work assignment;Revision No. 1 Date: October 2013 Section No. 2 Page 23 of 109

 Identification of any problems encountered;

 Identification of any deviations from the appli cable QAPPs (PQAPP, SQAPP, or project-specific QAPP); and

 Identification of problem resolution and/or correc tive actions taken during the reporting period, if
any.

Minor errors identified during the quality control checks described in this PQAPP will be addressed through the normal preparation and review pr ocess for a given deliverable. Significant quality deficiencies (pervasive errors or quality issues that could impact project budget and schedule) detected by technical reviewers or the ERG Project QA Coordina tor will be communicated, in writing, to the ERG WAM (with copies to the ERG Program Manager a nd ERG Contract QA Manager) for resolution. The ERG WAM will discuss significant quality deficienci es with the EPA WAM and is responsible for ensuring that action to correct the quality deficienci es is taken, documented, and reported to the ERG Project QA Coordinator.Revision No. 1 Date: October 2013 Section No. 3 Page 24 of 109

Toxic Weighted Pound Equivalents (TWPE) is calculated as the pounds of pollutant times the pollutant’s toxic weighting factor. Information about EPA’s Office of Water toxic weighting factors is available at: water.epa.gov/lawsregs/lawsguidance/ cwa/304m/upload/2005_09_19_guide _304m_2006_toxic_weighting.pdf and will be available in the 304(m) annual review records. 

### 3.0 C OLLECTION OF 

### E XISTING 

### D ATA FOR 

### U SE IN 

### C ONDUCTING 

### EAD’ S 

### A NNUAL 

### 304( M 

### ) I NDUSTRY 

### R EVIEW 

### P ROCESS

In support of the 304(m) annual review, ERG conducts environmental data collection, evaluation, and analysis activities on an alternating year schedule. EPA uses an odd- and even-numbered year approach to review industrial discharges according to the CWA, p ublishing a draft plan (for public comment) in oddnumbered years and a final plan in even-numbered years. Each year, ERG composes an Annual Review Report (ARR), which provides details on how data were generated, any data quality checks and corrections made, and the findings from analyses. Unde r this contract, ERG will continue to conduct oddand even-numbered year analyses and will prepare future AARs.

During odd-numbered years , ERG will conduct a toxicity rankings analysis (TRA) of all industrial discharges, including categories subject to existing EL Gs and discharges for industries with no categorical regulation. ERG will use the results from the TRA to prioritize the categories for further review based on toxicity of pollutant discharges ranked highest to lowest. To generate the TRA, ERG will use the EPA Office of Compliance’s DMR Pollutant Loading Tool 4 , available at http://cfpub.epa.gov/dmr/ , which provides category rankings by Toxic Weighted Pound Equivalents (TWPE). The DMR Pollutant Loading Tool uses discharge monitoring report (DMR) data and TRI data providing category rankings that can be drilled down to the facility and pollutant level.

The TRA focuses on identifying the “top industrial dischargers” of pollutants in effluent wastewater, expressed as TWPE. Top industrial dischar gers are generally defined as industries that collectively produce 95 percent of discharges in the United States, as measured in units of TWPE. As discussed in Section 3.3.1, ER G will conduct a review of the data for the top industrial dischargers, to identify errors and anomalies. After discussing a list of likely errors and anomalies with the EPA WAM, ERG will conduct follow-up calls as directed by the EPA WAM to specifi c facilities to verify the data inputs and obtain any necessary corrections.

ERG will prepare a table documenting the resu lting database changes, which will reference teleconferences and other backup data, such as permit s, verifying the validity of the database change. ERG will include this table in the ARR, so that all da ta corrections can be viewed by the public. ERG will work with EPA’s Office of Compliance to make co rrections to the underlying DMR Pollutant Loading Tool data so that future versions of the tool will include the updated, corrected data. ERG will continue these iterations of identifying data errors and correc tions and re-ranking industrial discharges by TWPE, as directed by the WAM.

From the corrected data, and at the EPA WAM’s di rection, ERG will perform further review of the discharges in industry categories with higher TW PE. ERG will investigate these industry sectors, including specific facilities, to understand the pro cess source of the discharge, the pollutants in the

4 DMR Pollutant Loading Tool Version 1.1 was released in November 2012.Revision No. 1 Date: October 2013 Section No. 3 Page 25 of 109

discharge, pollutant mass and pollutant concentrati ons. Using additional record information, including following up with facilities and/or pe rmit writers, engineering textbooks , scientific journal articles, and public comments on the previous EPA Preliminary or Final ELG Plans, ERG will support EPA in characterizing the source(s) of the pollutant discharg es. ERG will also identify and analyze categories or sectors with significant changes from the previous year (e.g., order of magnitude changes either higher or lower) in factors such as discharge amounts, cha nges in unit processes, advances in analytical method techniques, and pollution prevention advances or in novative technology usag e or availability. In particular, ERG will review categories acco rding to the following considerations:

 Amount of toxic pollutants discharged, whether large amounts of pollutants are discharged, and/or potential hazard to human health or the environment;

 Number of discharging facilities;

 Whether a demonstrated technology, process cha nge, or pollution prevention approach could be used to control pollutants;

 Cost, performance, and affordability of the tec hnology, process change, or pollution prevention measure; and

 Implementation and efficiency considerations.

The study of these categories or sectors will primarily focus on:

 Verification of TRI- and DMR-reported discharges;

 Preliminary identification of process source(s) of discharged pollutants; and

 Limited identification of potential control alternatives.

ERG will summarize its odd-numbered year review and prioritized categories in the ARR.

During even-numbered years , ERG will use additional data sources to supplement the TRA and identify unregulated industrial discharges not captured by the DMR and TRI data. Such data sources are described in Section 3.1.1.2. The even-numbered year review s will also seek to identif y and review wastewater discharges that are not quantified in DMR/TRI data or not prioritized based on TWPE, such as nutrients, contaminants of emerging concern, and conventional pollutants. Also, where more data or analyses are needed, as directed by the EPA WAM ERG will conti nue its reviews of categories identified during oddnumbered year reviews. ERG will s upport EPA’s end product of the even -numbered year review, the final ELG Plan, which publishes the results of the a nnual review and EPA’s schedule for any effluent guidelines development.

ERG will also perform additional in-depth “preliminary” or “detailed” studies or a somewhat less detailed “preliminary category review” in even-numbered years, as directed by the EPA WAM. In this way, EPA identifies existing categories for potential ELG re vision or new categories for ELG development.Revision No. 1 Date: October 2013 Section No. 3 Page 26 of 109



### 3.1 Data Gathering

To answer the key questions identified in Tabl e 2-2 in support of the preliminary and final ELG Plans, EPA needs to gather data characterizing industrial wastewater discharges and related treatment residues and data identifying the sources of the wastewater.



### 3.1.1 Data Sources

This section provides general descriptions of the da ta sources that ERG will use for the 304(m) annual reviews. Quality criteria and specifications for the co llected data are discussed in Section 3.3. ERG will obtain the existing data from databas es, publications, other EPA offices, www.regulations.gov , and other websites.



### 3.1.1.1 Data Sources for the TRA and Odd-Numbered Year Analysis

For the TRA and odd-numbered year analysis , ERG will use the following data sources:

 DMR Data . ERG will use DMR data reported for major dischargers and, where available, for minor dischargers. ERG will use permit fac ility data, parameter limits data, and measurement/violation data for the year 2011 (or later as available). The DMR Pollutant Loading Tool provides a report entitled Technical Users Background Document for the Discharge Monitoring Report (DMR) Pollutant Loading Tool , which describes the origin and maintenance of these datasets (available at http://cfpub.epa.gov/dmr/docs/T echnical_Users_Background_Doc.pdf ).

 TRI Data. ERG will use TRI data from the public release of reporting year 2011 (or later as available) to analyze TRI-reported releases to surface waters and POTWs. ERG will use data on chemical releases to water and off-site transfers to POTWs, the basis fo r release and transfer estimates, Chemical Abstracts Service (CAS) numbe rs, facility location and contact information, wastewater treatment information, dioxin distribut ion information, and releases to other media for select facilities.

 POTW Removal Percentages. ERG will use data from recently-developed pretreatment standards to identify chemical-specific removal rates typical of POTWs nationwide. The memorandum entitled “Publicly-Owned Treatment Works (P OTW) Percent Removals Used for the TRIReleases2002 Database” (Codding, 2005) describes the data sources used to identify POTW removal rates used in the database (see Att achment A). The memorandum entitled “Comments Received Regarding POTW Removals” (Bicknell, 2006) describes the comments received and the changes made to the POTW removals (see Attachment B). ERG will use the POTW removal rates in these memoranda (or subsequent updates to these memoranda) to estimate the quantity of chemicals transferred to POTWs that are ultimately discharged to receiving streams.

In addition, ERG will also re ference the following as needed:

 Point Source Category Crosswalk . ERG will rely on the Standard Industrial Classification (SIC) / North American Industry Classification System (NAICS) code to Point Source Category Crosswalk embedded in the DMR Pollutant Loading Tool.Revision No. 1 Date: October 2013 Section No. 3 Page 27 of 109

 CAS and Pollutant Parameter Groupings . For the point source category rankings, ERG will rely on the pollutant parameter groupings built into the DMR Pollutant Loading Tool. ERG would use the ungrouped parameter loadings only for facility-s pecific reviews or further review of specific industries. For example, ERG might evaluate if a facility reported dissolved metals instead of total metals when considering accuracy of loads or evaluating treatment technologies.



### 3.1.1.2 Additional Data Sources

At the direction of the EPA 304(m) project WAM, ERG identified data sources other than the DMR Pollutant Loading Tool to expand EPA’s ability to identify new pollutants of concern or wastewater streams in industrial categories not currently regul ated by ELGs. ERG will use these additional data sources (listed in Table 3-1) to research pollutants no t included in the TRA and also to identify improved wastewater treatment technologies.

Since 2003, EPA has relied on the TRA to support the annual reviews and planning process. EPA has also considered other data submitted to the Agency thro ugh the public review process. The TRA provides a national view of pollutant loadings discharged to surface water, identifies industries with the largest discharges, and highlights data errors in the unde rlying databases. However, this TRA process has consistently identified the same few industrial cat egories with high TWPE. The TRA’s usefulness in identifying categories is decreasing because it provides:

 Data for industries and pollutants that are alr eady regulated under NPDES permits or that are required to report to TRI. Therefore, new industri es, waste streams, and/or pollutants are difficult to identify with the TRA.

 No information on the state of the pollutant cont rol technology used to achieve such discharges; therefore, it is difficult to determine if pollutant discharges could be further reduced from current levels.

 No information on the economics of the industries discharging pollutants, which could be used to evaluate if they could afford more stringent controls.

Because of these limitations with the TRA, ERG iden tified, starting in 2012, additional data sources for the even-numbered annual reviews. ERG chose these sources seeking to identify new pollutants, pollution control technologies or alternatives, or new industri es of concern not already categorized or covered by the TRA. Table 3-1 lists the data sources iden tified and recommended by ERG for their potential relevance to the annual reviews.

Table 3-1. Additional Data Sources Identified for the Even-Numbered Annual Reviews

Data Source Relevant Content Data of Interest
Office of Air Quality Planning and Standards (OAQPS) air regulations: New Source Review (NSR), National Emission Standards for Hazardous Air Pollutants (NESHAPs), and New Source Air regulations may require air pollution controls that generate new wastewater streams for industrial categories. Treatment technology descriptions; performance data; technology costs.Revision No. 1 Date: October 2013 Section No. 3 Page 28 of 109

Table 3-1. Additional Data Sources Identified for the Even-Numbered Annual Reviews

Data Source Relevant Content Data of Interest

Performance Standards (NSPS)

Office of Water Targeted National Sewage Sludge Survey (TNSSS) Sewage sludge survey showing concentrations of pollutants in sewage sludge may link to industrial wastewater discharges. Discharge data, pollutants in sewage sludge, pollutant mass, pollutant concentration, flows.

Water Environment Federation technical journals Technical journal articles and presentations may identify wastewater treatment advances. Treatment technology descriptions, performance data, technology costs, sector trends (new facilities, processes, feedstocks).

Office for Chemical Safety and Pollution Prevention/ Office of Pollution Prevention and Toxic Substances (OCSPP/OPPTS), Significant New Use Rules, Chemical Data Reporting Rule (CDRR), and Chemical Action Plans Health and environmental hazard data for chemicals manufactured in or imported into the United States, which may or may not be regulated by ELGs. Also, new chemical manufacturing processes with unregulated discharges of new pollutants. Process descriptions, pollutants in discharges, environmental effects data, qualitative descriptions of impacts, human health effects data.

Office of Research and Development (ORD), National Risk Management Research Laboratory and New Analytical Methods Recent analytical methods that identify new pollutants of concern that are not already regulated by the ELGs (e.g., perflu orinated compounds and other contaminants of emerging concern). Pollutants that can be identified by the method.

2011 TRI Expansion Sectors (or subsequent updates) Supporting data for industrial sectors reviewed as part of the expansion may indicate pollutant discharges of concern. Pollutants in discharges, pollutant mass, pollutant concentration, flows.

Targeted stakeholder input in response to EPA outreach to selected industry groups EPA Regions, state and local regulatory agencies, industry, pretreatment coordinators, and citizen groups often collect data related to ELGs and industrial discharges outside of public comment. Pollutants in discharges, pollutant mass, pollutant concentration, flows, process descriptions, treatment technology descriptions, performance data, technology costs, sector trends (new facilities, processes, feedstocks).

Office of Groundwater and Drinking Water (OGWDW) OGWDW priorities that identify industries and contaminants of emerging concern, such as hydraulic fracturing, and emerging treatment technologies for drinking water that can be used to treat wastewater (identified in OGWDW’s Treatability Database). Treatment technology descriptions, process descriptions, performance data, technology costs.

U.S. Geological Survey (USGS), data warehouses and Publications on topics of emerging concern and USGS data warehouses that focus on ambient water quality but provide a potential means of Pollutants in discharges, impaired water descriptions.Revision No. 1 Date: October 2013 Section No. 3 Page 29 of 109

Table 3-1. Additional Data Sources Identified for the Even-Numbered Annual Reviews

Data Source Relevant Content Data of Interest publications prioritizing impaired waters and/or pollutants causing impairments.

Assessment and Watershed Protection Division, impaired waters data Impaired waters data to identify potential pollutants of concern. Pollutants in discharges, impaired water descriptions, pollutants in impaired waters.

Office of Water Standards and Health Protection Division, water quality criteria Water quality criteria currently under development to determine if any industries directly relate to a specific pollutant discharge. Water quality criteria.

Office of Enforcement and Compliance Assurance (OECA), enforcement initiatives Compliance initiatives may show new pollutants of concern or waste streams in industries that warrant review or revision. Pollutant in discharges, pollutant mass, sector trends, documentation of environmental impacts.

Public comments on the EPA 304(m) plans Comments on draft and final ELG plans often provide data. Pollutants in discharges, pollutant mass, pollutant concentration, flows, process descriptions, treatment technology descriptions, performance data, technology costs, sector trends (new facilities, processes, feedstocks).



### 3.1.2 Criteria for Selecting Data Sources and Resolving Data Gaps

EPA prioritized the review of data sources in Table 3-1 in 2012 and will consid er reviewing other data sources in future years. EPA has prioritized reviewin g data sources based on (1) likelihood of identifying unregulated discharges, and (2) utility in identifyi ng new wastewater treatment technologies or pollution prevention alternatives.

 Likelihood of Identifying Unregulated Industrial Discharges. Some of EPA’s program offices focus on contaminants of emerging concern that are not currently characterized by DMR and TRI data, such as the supporting documentation for OC SPP’s Chemical Action Pl ans for higher-chain perfluorinated compounds and the 2011 TRI Expansion Sector effort.

 Utility in identifying new wastewater treatment technologies or pollution prevention alternatives. For specific industries, EPA collects data to char acterize the performance a nd cost of applicable and demonstrated wastewater treatment technologies, process changes, or pollution prevention alternatives that could effectively reduce po llutant discharges during preliminary category reviews and detailed studies. For the 2012 revi ew, EPA selected data sources that report on treatment technologies and pollution prevention a lternatives across all industries. EPA’s reviews in future years may instead focu s on selected technologies or pollution prevention practices. ThisRevision No. 1 Date: October 2013 Section No. 3 Page 30 of 109

PQAPP will be revised or a SQAPP will be devel oped to incorporate this focus when, and if, it

occurs.

For the 2012 annual review, EPA selected the first sev en data sources on Table 3-1for ERG review. EPA will follow a similar rationale for future even-numbere d years, identifying a prioritized list of data sources in the odd-numbered year report that will receive public comment. Based on the odd-numbered year proposed list of sources, the EPA WAM will direct ERG on which sources to review and ERG will document the future data source selection in the ARR. ERG’s qualitative analyses will rely on these sources as well as published literature identified during review of materials related to the targeted analyses. The criteria that ERG will use to evaluate the quality of literature are: 1) accuracy, 2) reliability, and 3) representativeness.

 Accuracy . For purposes of this PQAPP, ERG assumes that the underlying data and information contained in state and federal reports and p eer-reviewed journal articles are accurate. ERG assumes that data from these sources are of suffici ent quality to identify po tential environmental impacts associated with indust rial discharges. ERG’s WAM will present data that are outliers (e.g., data that are not consistent with the body of information collected from the literature review) to the EPA WAM during their weekly stat us calls and will receive further direction on whether to incorporate or exclude the data in the analysis. Section 3.3.2 discusses data quality reviews for these data sources.

 Reliability. During the review of existing literature, ERG may encounter sources that contain limited documentation of the environmental im pact claim (e.g., unidentified speaker at a stakeholder meeting or a local resident commenti ng in a newspaper article). These data sources represent a lower level of reliability than peer-re viewed literature, but may still provide useful information on the range of environmental impact s associated with industrial discharges. If ERG identifies limitations in the quality of informa tion taken from other existing data sources (e.g., websites, newspapers, and stakeholder meeti ngs), the ERG WAM will communicate these data quality issues to the EPA WAM during their weekly status calls. ERG will also evaluate data reliability based on whether:

- Scientific work is clearly written, so that all assumptions and methodologies can be identified;

- Variability and uncertainty (quantitative and qualitative) in the information or in the procedures, measures, methods, or models are evaluated and characterized; and

- Assumptions and methodologies are consistently applied throughout the analysis as reported in the source.

 Representativeness. EPA requires data that have a nationa l perspective and are representative of an industry. ERG will evaluate representativen ess of existing data based on whether they represent:

- Types of environmental impacts caused by wastewater discharges;

- Range of concentrations; and

- Environments affected (e.g., rivers, lakes, and wetlands).Revision No. 1 Date: October 2013 Section No. 3 Page 31 of 109

ERG will use the criteria described in Section 3.3.2 below to identify if collected data are usable for this program. ERG will also evaluate the reasonableness of existing data. For example, pollutants must be reasonably related to the process or waste stream from which they were discharged. ERG may flag datasets as questionable and subject to further investigati on if a chemical is detected that is not expected in the waste stream or if an expected chemical is missing (based on historical data from earlier annual reviews). ERG will communicate to the EPA WAM whethe r reported or calculated pollutant loads, plant wastewater flows, or other existing data reflect the range of expected values, again based on historical data. In addition, if the data provided by a data s ource are repeatedly found to be unreasonable, ERG will discontinue use of that data source and communicate these quality issues to EPA.

ERG will not review the quality of existing data contained in publis hed EPA databases and data systems (e.g., TNSSS or other databases provided by federal or state agencies or trade organizations but will describe known limitations of the data in reports on wh ich they are based. If ERG identifies limitations in the quality of the data from published literature or fe deral/state generated databases that may impact decisions regarding the overall direction of further data collection efforts or the quality of deliverables, the ERG WAM will communicate these i ssues to the EPA WAM during their weekly status calls. Data limitation discussions will identify where data gaps exist and describe data reporting requirements.



### 3.1.3 Documentation and Records

ERG staff will use the following procedures when collecting published data and placing it in the project file:

 Collect full bibliographic information on all coll ected publications, including “date accessed” for information collected from the Internet;

 Save web pages as PDF files, so that the data s ource can be added to the electronic project file; and

 Use the following procedures when collecti ng data via telephone and email contact:

- Document all telephone contacts using a Telephone Contact Report depicted in Attachment C;

- Save all email contacts as PDF files, so that they can be added to the electronic project file; and

- Ensure that telephone contact reports and ema il records are added to the electronic project file.

Data gathered will be documented by the ERG project staff as source files in the project folder, which is maintained electronically on ERG’s project server. ERG’s WAM in consultation with the EPA WAM will determine which materials gathered during the 304(m) annual review process need to be included in the record that supports the ARR.



### 3.2 Data Management

ERG’s 304(m) Project File Management Plan, Revised June 2009 (ERG, 2009) is provided in Attachment D and presents detailed procedures that w ill be used for managing project files assembled in support of EPA’s preliminary and final ELG Plans and procedures for submitting appropriate portions ofRevision No. 1 Date: October 2013 Section No. 3 Page 32 of 109

these files to the docket. The ERG WAM assigns a Proj ect Assistant the responsibility of maintaining the file index and filing documents in the centralized proj ect file (including the electronic project file). The ERG WAM also ensures that project team members document all data sources used.

ERG also follows requirements for identification and c ontrol of different versions of the DMR Pollutant Loading Tool and any spreadsheet and/or database tables created to present and summarize collected data. These requirements include:

 Assigning a unique name and number (date or ve rsion number) to spreadsheets and/or databases created ;

 Maintaining a log that identifies spreadsheet a nd/or database revisions and describes the changes made to the different versions (log may be a data base table, a separate electronic file in the same folder as the database, or a file stored on removable media); and

 Maintaining a historical file of the DMR Polluta nt Loading Tool or spreadsheet/database used to support the annual review.

Section 7.0 presents details on ERG’s management of data in spreadsheets and Section 8.0 presents details on ERG’s management of data in project databases.



### 3.3 Data Quality Assessment

The primary objectives of the 304(m) annual data reviews are to ensure that:

 Data generated and collected are of the qua lity necessary for EPA to answer the questions identified in Table 2-2; and
Data analysis results are documented in the 304(m) pr oject file, and analyses are of the type, quality, transparency, and reproducibility needed.

Descriptions of data quality criteria and review procedures that ERG will use to identify data that are fit for this use are described in this section. ERG w ill collect information about industrial wastewater discharges from existing data sources. ERG anticipates collecting data on:

 Pollutant loads discharged by speci fied industries and/or facilities;

 Wastewater characteristics (e.g., volume, mass loads, pollutant concentrations);

 Industrial sector products and pollution control trends;

 Wastewater treatment technol ogy performance/efficiency;

 Wastewater treatment technology costs; and

 Environmental and human health effects of industrial wastewater discharges.



### 3.3.1 DMR Pollutant Loading Tool Data Quality Review and Verification

The review and evaluation of data pulled from the DMR Pollutant Loading Tool includes quality checks that occur within the loading tool programming and quality checks where ERG runs specific queries using the loading tool and/or examines accomp anying data, such as permit documents.Revision No. 1 Date: October 2013 Section No. 3 Page 33 of 109

EPA evaluates the quality of the DMR data for use in developing the current year’s loading database (e.g., DMR data for 2011 ) as part of the biennial TRA. This eval uation considers data completeness, accuracy, reasonableness, and comparability for facilities and industry categories ranked in the top 95 th percentile, based on TWPE. The Technical Users Background Document for the Discharge Monitoring Report (DMR) Pollutant Loading Tool (EPA, 2012) documents the built-in quality checks, which include the load calculator routines. (This document is available at htpp://cfpub.epa.gov/dmr/docs/Tec hnical_Users_Background_Doc.pdf ).

The following discussion describes additional quality re view steps that ERG will perform at the EPA WAM’s direction:

 Completeness, reasonableness, and co mparability of DMR and TRI data. Using data in the DMR Pollutant Loading Tool, ERG will compare DMR and TRI data from different years to evaluate its completeness, reasonableness, and comparability . For example, as part of the 2012 304(m) annual review, ERG will use the DMR Pollutant Loading Tool to compare the number of facilities and the universe of SIC and NAICS codes in the 2011 DMR and TRI data to the data in 2009. ERG will also compare the number of facilities reporting, increases and/or decreases in loads, and pollutants reported for the top 95 th percentile.

 DMR Pollutant Loading Tool results. ERG’s quality review of the DMR Pollutant Loading Tool results includes the following checks. Each of these checks may result in a list of suspected inaccuracies. At the EPA WAMs direction, ERG will contact the impacted facilities and/or permitting authority for clarification. Based on these reviews, ERG will generate a list of identified inaccuracies and will review the list w ith the EPA WAM for inclusion or exclusion. At the EPA WAM’s direction, ERG will record the co rrection needed in a database corrections table that is included with the ARR as part of the 304( m) project record. ERG also delivers the entire list of database corrections to EPA’s Office of Comp liance for inclusion in subsequent versions of the DMR Pollutant Loading Tool.

- Accuracy of Facility Discharges . ERG will review and verify the accuracy of facilities’ discharges that had the greatest impact, in terms of TWPE, on total point source category loads and point source category rankings to identify possible calculation errors. ERG reviews the monitoring period and measurement data available in the tool for the top 95 th percentile facilities to information from each facility’s NPDES permit and permit fact sheet to see if they match. For discharge amounts, ERG will compare those reported the DMR Pollutant Loading Tool to the permit limits for the polluta nt to identify values that are an order of magnitude different since this is often an indication of a units error.

- Accuracy of Point Source Category Discharges . ERG reviews for accuracy point source category discharges by verifying that pollutant discharges in the DMR Pollutant Loading Tool (for the top 95 th percentile of facilities in the point source) were assigned to the appropriate point source category. ERG researc hes facility data using permit fact sheets, engineering textbooks and encyclopedias (such as Kirk-Othmer), and facility websites. ERG uses engineering judgment to determine if the pollutant discharge was reasonably associated with the point source category.

- Reasonableness of Pollutant Loads . ERG reviews the output from the DMR Pollutant Loading Tool for those pollutant discharges with the highest TWPE (e.g., dioxins, polychlorinated biphenyls [PCBs], and mercury) to identify possible erro rs in recording unitsRevision No. 1 Date: October 2013 Section No. 3 Page 34 of 109

of measure. For these top pollutants, ERG identif ies calculated discharges (at the facility and discharge pipe point) that are orders of magnit ude higher than previous years’ discharges or discharges from other facilities within the sa me point source category. ERG then reviews quantities or concentrations and flows that the DMR Pollutant Loading Tool used to calculate the annual discharge. The DMR Pollutant Load ing Tool includes a function to compare these measurements with measurements availabl e in EPA’s Enforcement Compliance History Online (ECHO) web page. ERG also compares the data to permit documents. If the measurements are similar, ERG concludes that the output is acceptable. If the data do not match between the databases and ECHO, the data are noted as an inaccuracy to be reviewed with the EPA WAM.

- Reasonableness of Facility Loads . ERG identifies facilities for review by whose pollutant discharges account for more than 95 percent of the TWPE for the point source category. ERG compares the facility’s DMR data to other av ailable information, such as the facility’s NPDES permit and permit fact sheet and inform ation from ECHO. If the data do not match between the database and ECHO, ERG will identify it as an inaccuracy to be reviewed with the EPA WAM.

- Comparability. ERG compares the DMR data to prior years of data (such as the comparison of 2011 year data to 2009 year data) to identif y pollutant discharges or wastewater flows that differed more than the year-to-year variation of other chemicals and facilities. ERG uses this comparison to determine if quantity, concentrati on, or flow corrections are needed for facility discharges with the highest TWPE. If the compar ison is unavailable (e.g., the pollutant was not previously reported), ER G will contact the facility or permitting authority to solicit the needed data.

 Review of Database Corrections from Past Years. ERG will review database corrections for past years (e.g., DMRLoads2009, DMRLoads2008, DMRLoads2007
, PCSLoads2004
, PCSLoads2002, and PCSLoads2000 ) to determine whether corrections made during previous reviews should apply to the current year DMR discharges.

 Review of Current DMR Data. ERG will compare hand-calculated annual pollutant load results to loads calculated by the DMR Pollutant Loading Tool.

 Review of Monitored Pollutant Discharges. ERG will review DMR pipe description information, ECHO discharge data, and the fac ility’s NPDES permit and permit fact sheet to identify monitored pollutant discharges that are:

- Intermittent (e.g., occur after periodic operati ons, seasonal, or occur after a storm event);

- Internal monitoring locations from which w astewater is combined with other waste streams and monitored again, resu lting in double-counting loads; and

- Not representative of point source category discharges (e.g., stormwater runoff from nonprocess areas, noncontact cooling water, or wast ewater related to operations in another point source category).Revision No. 1 Date: October 2013 Section No. 3 Page 35 of 109



### 3.3.2 Other Existing Data Quality Review and Verification

The criteria ERG will use to review and evaluate th e quality of non-DMR existing data are summarized in Table 3-2, along with acceptance criteria based on how the data will be used. ERG will use other existing data on the 304 (m) project for the following uses: identifying industrial sector trends, describing wastewater characteristics, describing treatment system performance, estimating pollutant loads, estimating treatment costs, and describing human heal th and environmental impacts. Existing data sources will include discharge data (e.g., data received durin g stakeholder outreach and databases from other EPA offices, such as CDRR and TNSSS), articles and tech nical papers, industry/vendor data, and Internet information. Table 3-2 provides the specific circumstan ces that define when data meet acceptance criteria or when data are unusable.

Data verification is the confirmation by examination and review of objective evidence that the data are complete, correct, consistent, and in compliance with technical requirements, es tablished standards, and contractual requirements. To accomplish this, ERG will check the data source and/or data itself at the time of data collection against the quality criteria of Table 3-2. The ERG WAM will discuss any quality concerns from these reviews with the EPA WAM dur ing their weekly status calls. ERG will also document in the project technical reports if any data or data sources do not meet the criteria.

ERG may use data that do not meet the specifications if, for example, there are no other available data. ERG will discuss such data quality issues with the EPA WAM and obtain written approval before using these data in subsequent analyses. This approval will be added to the 304(m) project quality file. In the annual ARRs that support the annual preliminary or final effluent guidelines program plans, ERG will describe data quality, including any need to use ot her existing data that did not meet the quality acceptance specifications in Table 3-2.



### 3.3.3 Achieving the Contract’s Overall Quality Objectives

ERG’s procedures described in Section 3.1.1 result in iden tification of a variety of existing data of interest to EPA from numerous sources. ERG’s systematic re view of those data against the data quality specifications presented in Table 3-2 help to ensure that the data are of a known and acceptable quality. Through these procedures, the existing data identif ied and collected for use will be unbiased and the resulting dataset used by EPA will meet EPA’s goal of objectivity. ERG’s procedures to ensure the security of existing data collected are described in Section 3.2. In addition, ERG’s procedures for the handling of existing data in spreadsheets and databases (presented in Section 7.0 and 8.0 below) helps to ensure data integrity and that data are not improperly altered during the collection and/or analysis process. The objective of collecting existing data that is usab le is accomplished through ERG’s application of this section’s procedures, including methods for identifying usable sources of data and evaluation of its quality in consultation with the EPA WAM.Revision No. 1Date: October 2013Section No. 3Page 36 of 109Table 3-2. 304(m) Annual ReviewData Sources Acceptance CriteriaType of DataData Not UsableData Usable forIndustrial SectorTrendsData Usable toCharacterizeWastewaterData Usable toDemonstrateTreatment SystemPerformance/EfficiencyData Usable toEstimatePollutant LoadsData Usable toEstimate Cost ofTreatmentData Usable forEnvironmentaland HumanHealth EffectsDischarge Data(e.g., databasesfrom other EPAoffices such asWATERS)(1) Data collectedby an unknownmethod or unitsundefined.(2) Data collectedduring upsetconditions.(3) Represents aprocess that is notof interest (e.g.,sanitarywastewater).Waste streamidentified andanalytesidentified.Waste streamidentified andanalytes, units, anddetection limitsidentified.(1) Represents full-scalesystem.(2) Paired influent andeffluent data show thattreatment system is welldesigned and operated.(3) Analytes identifiedand units included.Identified asdischarge,analytesidentified andunits provided,approvedanalyticalmethod used,represent currenttreatmentoperations.Discharge dataare not typicallyused for thispurpose.Waste and typeof receivingstream identifiedand analytesidentified.Articles /Technical PapersInformation doesnot address Table2-2 key questionsDocumented andreliable sourcesfound throughliterature searches(e.g., peerreviewed journalsand governmentpublications).Waste streamidentified andanalytes, units, anddetection limitsidentified.(1) Represents full-scalesystem.(2) Paired influent andeffluent data show thattreatment system is welldesigned and operated.(3) Data do not representupset conditions.(4) Analytes identifiedand units and detectionlimits included.Identified asdischarge,analytesidentified andunits provided,approvedanalyticalmethod used,representstreatmentoperations.Cost componentsfully defined andidentified (e.g.,direct andindirect capital,annual operatingandmaintenance),system size andcost yeardefined.Documented andreliable sourcesfound throughliteraturesearches (e.g.,peer-reviewedjournals andgovernmentpublications) thataddress keyquestions.Revision No. 1Date: October 2013Section No. 3Page 37 of 109Table 3-2. 304(m) Annual ReviewData Sources Acceptance CriteriaType of DataData Not UsableData Usable forIndustrial SectorTrendsData Usable toCharacterizeWastewaterData Usable toDemonstrateTreatment SystemPerformance/EfficiencyData Usable toEstimatePollutant LoadsData Usable toEstimate Cost ofTreatmentData Usable forEnvironmentaland HumanHealth EffectsIndustry /Vendor DataInformation doesnot address Table2-2 key questions.These data aregenerallyacceptable forrelevant sources.Data not typicallyused for thispurpose.(1) Represents full-scalesystem.(2) Paired influent andeffluent data show thattreatment system is welldesigned and operated.(3) Data do not representupset conditions.(4) Analytes identifiedand units and detectionlimits included.Data nottypically used forthis purpose.Cost componentsfully defined andidentified (e.g.,direct andindirect capital,annual operatingandmaintenance),system size andcost yearidentified.These data aregenerallyacceptable forrelevant sources.InternetInformation(other thanpublishedarticles ortechnical papers)Information doesnot address Table2-2 key questions.These data aregenerallyacceptable forrelevant sources.Data not typicallyused for thispurpose.Data not typically usedfor this purpose.Data nottypically used forthis purpose.Data nottypically used forthis purpose.These data aregenerallyacceptable forrelevant sources.Revision No. 1 Date: October 2013 Section No. 4 Page 38 of 109

Characterizing process wastestreams may include identifying the frequency and volume of generation, flowrates, pollutants in the wastestream, and pollutant concentrations.

Characterizing wastewater treatment system performance may include identification of influent and effluent pollutant concentrations and/or loads and the percent removal of pollutants through the system. 

### 4.0 C OLLECTION OF 

### E XISTING 

### D ATA FOR 

### U SE IN 

### EAD’ S 

### I NDUSTRY 

### S TUDIES AND 

### ELG D EVELOPMENT

ERG collects existing data in support of all of EAD’ s programs. Section 3.0 of this plan discusses existing data use for the 304(m) annual review process, while this section addresses existing data use in support of EAD’s studies and/or ELG development. Collection of existing data is necessary for the following

reasons:

 EAD programs require a variety of industry, pro cess, treatment, and environmental-related data, not all of which is available throug h primary data collection activities;

 Collection of existing data allows EPA to make more effective use of limited resources (in both time and budget) than collection of primary data alone; and

 Existing data may provide qualitative and/or qu antitative information that supports EAD’s primary data collection activities.

This section details ERG’s procedures for (1) identifyi ng and selecting existing data for use in support of EAD programs and (2) QA/QC that ERG follows to ev aluate and document the quality of existing data collected and to describe the limitations of its use.



### 4.1 Data Gathering

ERG and EPA use data collected from existing s ources to support EAD’s industry studies and/or development of ELGs. These data are documented in study reports and public records for rulemaking activities and are subject to public review and comment. This section discusses ERG’s procedures for gathering existing data in support of EPA’s industry studies and ELG development programs.



### 4.1.1 Data Needs and Potential Data Sources

ERG begins gathering existing data first by worki ng with EPA to define data needs for the work assignment/project and the intended use of the da ta. Data needs are identified by the EPA work assignment, ERG’s description of our technical approach in the work plan, through meetings and discussions, and through planning memoranda. As dir ected by the EPA WAM, ERG will prepare data collection memoranda to document the task specific a pproach for existing data collection. Under Contract EP-C-12-021, ERG will collect existing data to:

 Create, revise, or support an industry profile;

 Identify and characterize process wastestreams;

 Identify applicable treatment technologies and/or pollution prevention practices for specific waste streams and/or pollutants of concern;

 Characterize wastewater treatment system performance/efficiency;

 Estimate baseline and option pollutant discharges;

 Estimate treatment costs;Revision No. 1 Date: October 2013 Section No. 4 Page 39 of 109

 Identify environmental impacts associat ed with pollutant discharges; and

 Estimate water column, sediment, and fish tissue pollutant concentrations.

EPA can then use the data to iden tify and evaluate alternative regulat ory approaches. Table 4-1 presents the data elements that ERG will collect for the listed existing data purpo ses and potential sources for them.

Table 4-1. Existing Data Purpo ses and Data Elements Gathered

Existing Data Purpose Data Elements Gathered Potential Data Sources
Create, Revise, or Support An Industry Profile  Facilities in the universe of a specified point source category.

 Facility location information.

 Facility ownership information.

 Facility size.

 Facility operation descriptions (may be based on type of fuel use, type of product, type of operations).

 Facility financial information (publicly available).  Existing agency databases.

 Other governmental agency databases/lists.

 Trade association lists.

 Dun and Bradstreet.

 Company/facility publications and websites.

 Technical literature.

 Company financial statements.

Identify and Characterize Process Wastestreams  Industry descriptions.

 Facility-level operation descriptions.

 Process wastestream pollutants and concentrations.  Existing agency documents (e.g., sector notebooks, background documents, study reports).

 Agency databases (e.g., DMR and TRI datasets).

 Other governmental agency documents.

 Technical literature.

 Existing industry/facility site visit and sampling data.
Identify Applicable Treatment Technologies and/or Pollution Prevention Practices for Specific Waste Streams and/or Pollutants of Concern  Industrial treatment technology descriptions.

 Performance data on specific technologies or pollution prevention practices.

 Pollution prevention practice descriptions by industry/operation.  Existing Agency documents and databases (e.g., study reports, NRMRL treatment performance database).

 Other governmental agency documents and databases (e.g., TRI database, ATSDR studies).

 Technical literature.

 Conference papers.

 Trade association publications.Revision No. 1 Date: October 2013 Section No. 4 Page 40 of 109

Table 4-1. Existing Data Purpo ses and Data Elements Gathered

Existing Data Purpose Data Elements Gathered Potential Data Sources

Characterize Wastewater Treatment System Performance/Efficiency  Treatment performance data including influent and effluent concentrations.

 Percent reduction through treatment for a specified pollutant.

 Analytical methods used.

 Sampling and analytical QA/QC performed.  Existing Agency sampling data.

 Existing Agency treatability studies.

 NRMRL’s Treatability database.

 Existing industry/facility sampling data.

 Technical literature.

 Vendor discussions/data/studies.

Estimate Baseline and Option Pollutant Discharges  Industry effluent concentrations and flows.

 Facility-specific effluent concentrations and flows.  Existing Agency sampling data on facility/industry effluent concentrations and flows.

 Agency databases (e.g., DMR and TRI datasets).

 Existing industry/facility sampling data on effluent concentrations and flows.

 Technical literature.

 Vendor discussions/data/studies.

Estimate Treatment Costs Capital and/or annual costs for a specific technology, technology train, or pollution prevention practice.  Vendor cost data.

 U.S. Army Corp of Engineers CAPDET model.

 RS Means Construction Cost data.

 Technical literature.

Environmental Assessment Analysis  Pollutant toxicity.

 Human health effects data.

 Aquatic health effects data.

 Linkage of facility to its discharge water body.

 Receiving stream characteristics.

 Documentation of environmental incidents/harm.  Agency studies and reports.

 Agency databases (e.g., WATERs, 303(d) list of impaired waters).

 USGS databases.

 National Hydrography Dataset.

 Existing agency toxicity databases.

 Other agency studies and databases.

 Technical literature.

 GIS coordinates for facilities and water bodies.

 Conference proceedings.

 Gray sources (such as newspaper articles or press releases).

ERG identifies potential data sources through:

 Identification by the EPA WAM and/or other Agency staff;

 Identification by industry or other agency contacts;Revision No. 1 Date: October 2013 Section No. 4 Page 41 of 109

 Review of references cited in previo usly identified published literature; and

 Literature search.

Section 4.1.2 below presents details on ERG’s pr ocedures for conducting literature searches and the search engines that we use to identify existing data sources. As directed by the EPA WAMs, ERG will document literature search methodology and results in me moranda or literature search review reports that specify data needs, key words used, sources identified, and ERG’ s review of the source quality.



### 4.1.2 Literature Searches

ERG literature searches use a keyword list along with a search engine to identify technical papers/reports, studies, and peer-reviewed journal articles that documen t and/or provide data on a specific technical topic. In addition, as time is available, ERG will search for information from newspapers, environmental groups, industry organizations, and other non-peer re viewed information sources. ERG acknowledges that EPA considers these sources “gray” literature that ma y not be acceptable forms of formal documentation. However, ERG recognizes that these literature sources can provide useful information for identifying incidental and qualitative data and examples that s upport data from peer-reviewed information sources. For example, in conducting literatu re searches for environmental ass essment analyses, ERG will often find environmental events reported in gray literature sources before an environmental impact is well documented in peer-reviewed journals or government re ports. ERG intends to use gray literature to help highlight areas of interest and fac ilitate additional searches of peer-rev iewed journals for more detailed information on the topic of interest. EPA may also di rect ERG to use gray literature if other preferred sources of data are not available.

ERG uses several different search engines to broaden th e range of reference materials represented in the results. These search engines include:

 Dialog ( www.dialog.com, a licensed search product) – Dialog is a comprehensive search engine for searching articles and reports from thousands of in-depth repositories of scientific and technical data, patents, trademarks and other in tellectual property data, real-time news feeds, newspapers, broadcast transcripts and trade publica tions, plus market research reports and analyst notes providing support for financial decision-making.

 Scirus ( www.scirus.com ) – Scirus is a comprehensive science-specific search engine. Scirus provides access to a large database of scientific, technical, and medical journals.

 Ingenta ( www.ingentaconnect.com ) – Ingenta is a scholarly research database that provides access to a large database of academic and professional research articles.

 Google Scholar ( http://scholar.google.com/ ) – Google Scholar is a general search engine of scholarly literature, including peer-reviewed pa pers, theses, books, abstracts and articles from academic publishers, professional societies, prep rint repositories, and universities and other scholarly organizations.

 Google ( www.google.com ) – Google is a general search engine of information readily available on the Internet.Revision No. 1 Date: October 2013 Section No. 4 Page 42 of 109

ERG will use a key word list developed for the proj ect and will often pair multiple keywords to focus search results. ERG develops multiple keyword combin ations using our best professional judgment and knowledge of the topic under investigation. ER G will document key words in a project planning document and/or in the literature review documentati on as described in Section 4.1.5, unless directed by EPA not to do so.



### 4.1.3 Criteria for Selecting Data Sources and Resolving Data Gaps

Criteria that EPA and ERG use to eval uate candidate data sources include:

 Relevance of the source to the data need;

 Reliability of the source; and

 Content of the source.

Relevance. Data sources specific to the topic being investigated or containing data specific to the industry, operation, wastestream, pollutant of interest, and/or receiving water will be considered for use. Sources that most closely represent the topic/data of interest are the most relevant.

Reliability. ERG and EPA consider the following factors when evaluating reliability of a data source: (1) sources that have been generated by EPA and/or other governmental agencies and subject to data review and assessment are considered to be the most reliabl e, (2) sources generated by entities with established knowledge in the topic area (e.g., studies conducted by in dustry experts, primary data generated by an industrial facility using documented and approved methods) are also cons idered by EPA to be reliable, and (3) sources that use unknown collection and data review procedures are considered by EPA as less reliable.

Content . EPA also considers the source’s content in sel ecting it for use. Examples of source content can include extent of data (e.g., how many facilities do es it cover, over what geographical area, other what period of time), age of the data, and level of doc umentation describing the generation of the data.

EPA and ERG will meet to discuss the identified existing data sources for each project. Through discussion, EPA and ERG will decide whether suffici ent relevant and reliable data sources exist and whether those sources contain sufficient data for the intended use. The gathering of existing data in support of EAD studies and ELGs is an iterative pro cess and will often involve the following steps:

 Identification of existing data sources;

Assessment of their releva nce, reliability, and content;

Determination of whethe r data gaps exist; and
Resolutions for filling data gaps.

To resolve data gaps, EPA may instruct ERG to iden tify additional existing data sources, collect primary data, and/or develop assumptions or a methodology to address the data gap.Revision No. 1 Date: October 2013 Section No. 4 Page 43 of 109



### 4.1.4 Data Value Selection Approach

As part of the data source selec tion and data gap evaluation described in Section 4.1.3, ERG will work with EPA to conduct data value selection, i.e., the selec tion of data elements of interest from the potential sources identified. Once ERG and EPA have screened th e potential sources of needed data, we may find that:

 More than one source offers the data element(s) of interest;

 Only one source provides the data element of interest; or

 No sources yield the data elements of interest.

When more than one source offers the data element( s) of interest, the following data selection approaches are used:

 If only one of the sources is deem ed relevant and reliable, then that source will be used;

 If more than one of the sources is deemed releva nt and reliable, ERG will examine the range of values and will recommend an approach. EPA can then choose to average the data values from the multiple acceptable sources or can select a pr eferred source (where preference is given to values that best fit the data need); and

 If none of the sources are deemed relevant and re liable, EPA may direct ERG to conduct further literature searches or to develop an assumption or methodology to estimate the needed value.

When values are used from multiple existing data sour ces and when directed to do so by EPA, ERG will investigate the range of data values found, whether th e data represent normal distributions, or whether the data appear to include non-normal distribution of values. In cases where multiple data points are normally distributed, averaging the data for an arithmetic m ean is appropriate. In cases where multiple data points are non-normal in their distribution, then using a median of the data may be more appropriate. Finally, in cases where a single value is not required for the anal ysis, presentation of the value range may be most appropriate. ERG will work with the EPA WAM in deciding which approach to use for multiple acceptable values.

When only one source offers the data element(s) of interest, the following data selection approaches are used:

 If the source is deemed relevant and re liable, that source will be used; and

 If the source is not deemed relevant and relia ble, EPA may direct ERG to conduct further literature searches or develop an assumption or methodology to develop the needed data.

When none of the potential data sources yield the data elements of interest, EPA may direct ERG to:

 Conduct additional literature searches for a relevant and reliable source of the data element(s);

 Collect needed data element(s) directly throug h primary data collection procedures (e.g., through a phone contact or sampling); or

 Develop an assumption or methodology to develop the needed data.Revision No. 1 Date: October 2013 Section No. 4 Page 44 of 109



### 4.1.5 Data Gathering Documentation and Records

The EPA WAM will determine the manner and format for literature review documentation. ERG will document the methodology used for identifying existing da ta sources, the criteria used in selecting data sources for further use, and the methodology for data selection in planning memoranda, data search and result reports, and final deliverables unl ess directed by EPA not to do so.

ERG will obtain data from sources such as databas es, publications, other EPA offices, government publications, and scientific literature.

ERG staff will use the following pro cedures when collecting published data:

 Collect full bibliographic information on all coll ected publications, including “date accessed” for information collected from the Internet.

 Save web pages as PDF files, so that the data s ource can be added to the electronic project file.

ERG staff will use the following documentation proce dures when collecting data via telephone and email

contact.

 Document all telephone contacts using a Telephone Contact Report depicted in Attachment C.

 Save all email contacts as PDF files, so that they can be added to the electronic project file.

 Ensure that telephone contact repor ts and email records are added to the electronic project file.

Data gathered will be documented by ERG project staff and saved as files in the project folder maintained electronically on ERG’s project server. ERG’s WAM in consultation with the EPA WAM will determine which data materials gathered during the project need to be included in any reco rd that supports the study and/or ELG.



### 4.2 Data Management

Specific projects may have a project file manageme nt plan that presents detailed procedures ERG will use for managing the project files assembled in su pport of the project and for submitting appropriate portions of these files to a docket (if applicable). For example, all engineering information collected, generated, and analyzed on the steam electric ELG pr oject is handled and stored in a centralized record, as described in the Steam Electric Power Generating Project File Management Plan , dated September 25, 2009. This centralized record contains all data collected or accessed from other sources and documents analyses of data collected (e.g., site visit reports, memoranda, industry profiles). The contents of the record are indexed using a MS Access® database.

For projects without a specific file management plan, the ERG Program Manager and ERG WAM are responsible for ensuring project documents and reco rds conform to technical requirements and quality system requirements specified in the work assignmen t or technical direction by the EPA WAM. Work assignment planning documents (e.g., the work plan and this PQAPP) identify the level of review required for project deliverables and who will conduct the review. This review is conducted using the quality assessment tools described in this PQAPP. The ERG Program Manager and ERG WAM are also responsible for ensuring that records and docum ents accurately reflect the completed work.Revision No. 1 Date: October 2013 Section No. 4 Page 45 of 109

For all EAD work assignments that ERG conducts under Contract EP-C-12-021, the ERG WAM assigns a Project Assistant the responsibility of maintaining the project file index and filing documents in a centralized project file (including the electronic project file). For non-EAD projects, project files may not be needed; if a non-EAD project requires a project file, the ERG WAM will assign project staff to create and maintain one. Stored records (hardcopy) are pr otected from damage, loss, and deterioration at all ERG locations by storing them on site in locked, climate-controlled file rooms. All ERG offices are equipped with smoke alarms and sprinkler systems in case of fire. Access to the record storage areas is limited to authorized personnel. Electronic project data, calculations, programs, models, databases, and written products are stored in identified project areas on ERG’s network. ERG conducts daily tape backups to protect against loss of electronic data. If a network server is damages or destroyed, electronic files will be recovered from back-up tapes and a ne w network server will be established. ERG makes back-up tapes of all server files on a weekly basis and stores them as follows: full back-up from the end of the previous week is stored in each office in a fi re- and water-proof safe; full back-up from two weeks prior is stored in an off-site location. The ER G WAM will ensure that project team members are instructed on use of that project’s file management system.

ERG procedures for the entry and/or uploading of el ectronic data from existing sources into spreadsheets and databases are further discussed in Sections 7.0 and 8.0.

For Contract EP-C-12-021, ERG also stores inactive pa per records in on-site storage rooms. Inactive electronic files are archived to tape or compact disk for long-term storage. ERG maintains the project files for EPA’s Office of Water projects until the materials can be: 1) boxed and sent to the Federal Records Center (FRC), 2) returned to EPA’s offices as directed , or 3) destroyed if directed to do so by EPA. Documents obtained from clients or other entities, or created by ERG, that fall under CWA CBI requirements are destroyed or retu rned to the appropriate person(s) in accordance with the entities’ written instructions. Other classified or confidential re cords are destroyed by sh redding or incineration.



### 4.3 Existing Data Quality Assessment

After identification and selection of existing data s ources, EPA and ERG then review the data in those sources to assess its usability. The subsections below present and discuss quality criteria for existing data and how ERG will assess whether it is suitable for use.



### 4.3.1 Quality Review of Existing Data Used in Characterizing Industry Point Sources, Wastestreams, and Treatment Technologies

The criteria that ERG will use to evaluate the quality of literature sources used for characterizing industry point sources, wastestreams, and/or treatment t echnologies are: (1) accuracy, (2) reliability, and (3) representativeness.

 Accuracy . ERG assumes that the underlying data and in formation contained in state and federal reports and peer-reviewed journal articles are a ccurate. ERG assumes that data from these sources are of sufficient quality to support industry and/or wastestream characterization and the performance of specified treatment technologies. ERG will present data that are outliers (e.g., data that are not consistent with the body of information collected from the literature review) to EPA for further direction on how to incorpor ate or exclude the data in the analysis.Revision No. 1 Date: October 2013 Section No. 4 Page 46 of 109

 Reliability. During the course of a literature search, ERG may encounter sources that contain limited documentation of the source of data/info rmation (e.g., non-peer reviewed industry publication). These data sources represent a lowe r level of reliability than peer-reviewed literature, but may provide useful information for capturing the full range of processes and/or wastes generated by a specified industry point source or a range of information on specified treatment technologies. If ERG identifies limita tions in the quality of information taken from other existing data sources (e.g., trade publi cations, vendor literature), ERG will communicate these data quality issues to EPA. ERG will also eval uate existing data for reliability based on the following factors:

- The literature is clearly written, so that a ll assumptions and methodologies can be identified;

- The variability and uncertainty (quantitative and qualitative) in the information or in the procedures, measures, or results are evaluated and characterized; and

- The assumptions and methodologies are consiste ntly applied throughout the analysis as reported in the source.

 Representativeness. ERG will also evaluate the representativ eness of existing data to ensure that it comes from the technology or process of inter est. For example, pollutant identities must represent the process or wastestream from which they were discharged. ERG may flag a data set as questionable and subject it to further investigation if a detected chemical or its concentration is not expected in a waste stream or if an expect ed chemical is missing. ERG will communicate to the EPA WAM whether reported or calculated pollutant loads, plant wastewater flows, or other existing data reflect the range of expected va lues for specific operations/waste streams in the industry of concern. In addition, if the data provi ded by a data source are repeatedly found to be unrepresentative, ERG will discontinue use of th e source and communicate these quality issues to EPA..

When existing data are collected, ERG will use the criteria in Table 4-2 to recommend the types of analyses existing data can be used for, if any.Revision No. 1Date: October 2013Section 4Page 47 of 83Table 4-2. Existing Data Acceptance Criteria for Industry Studies and ELG DevelopmentType of DataData Not UsableData Usablefor ProfileData Usable toCharacterize InProcess WasteData Usable toDemonstrate SystemPerformance/EfficiencyData Usable toEstimate BaselineDischargesData Usable toEstimate Cost ofTreatmentMonitoring Data(e.g., NPDESpermit, DMRs)(1) The plant hassince changedoperations (e.g.,installed a newtreatment system)since the data werecollected.(2) Data collectedduring upsetconditions.Waste streamidentified andanalytes identified.Waste streamidentified andanalytes, units,and detectionlimits identified.(1) Represent full-scale systemoperated at applicable steamelectric power plant.(2) Influent and effluent data orpercent removal identified.(3) Analytes identified and unitsincluded.Identified asdischarge, analytesidentified and unitsprovided, approvedanalytical methodused, representcurrent treatmentoperations.Data not typicallyused for thispurpose.Other MonitoringData (e.g., 308Letter,Questionnaire,EPRI, In-Plant)(1) Data collectedby an unknownmethod or unitsundefined.(2) Data collectedduring upsetconditions.(3) Represents aprocess that is notof interest (e.g.,sanitarywastewater).Waste streamidentified andanalytes identified.Waste streamidentified andanalytes, units,and detectionlimits identified.(1) Represents full-scale systemoperated at applicable steamelectric power plant.(2) Influent and effluent datashow that treatment system iswell designed and operated.(3) Analytes identified and unitsincluded.Identified asdischarge, analytesidentified and unitsprovided, approvedanalytical methodused, representcurrent treatmentoperations.Cost componentsfully defined andidentified (e.g.,direct and indirectcapital, annualoperating andmaintenance), costyear identified.Revision No. 1Date: October 2013Section No. 4Page 48 of 109Table 4-2. Existing Data Acceptance Criteria for Industry Studies and ELG DevelopmentType of DataData Not UsableData Usablefor ProfileData Usable toCharacterize InProcess WasteData Usable toDemonstrate SystemPerformance/EfficiencyData Usable toEstimate BaselineDischargesData Usable toEstimate Cost ofTreatmentArticles/TechnicalPapersInformation doesnot address Table2-2 key questionsDocumented andreputable sourcesfound throughliterature searches(e.g., peerreviewed journalsand governmentpublications).Waste streamidentified andanalytes, units,and detectionlimits identified.(1) Represents full-scale systemoperated at applicable steamelectric power plant.(2) Influent and effluent datashow that treatment system iswell designed and operated.(3) Data do not represent upsetconditions.(4) Analytes identified and unitsand detection limits included.Identified asdischarge, analytesidentified and unitsprovided, approvedanalytical methodused, representtreatment operations.Cost componentsfully defined andidentified (e.g.,direct and indirectcapital, annualoperating andmaintenance), costyear identified.Industry / VendorTelephone QueriesInformation doesnot address Table2-2 key questions.These data aregenerallyacceptable forrelevant sources.Data not typicallyused for thispurpose.(1) Represents full-scale systemoperated at applicable steamelectric power plant.(2) Influent and effluent datashows that treatment system iswell-designed and operated.(3) Data do not represent upsetconditions.(4) Analytes identified and unitsand detection limits included.Data not typicallyused for this purpose.Cost componentsfully defined andidentified (e.g.,direct and indirectcapital, annualoperating andmaintenance), costyear identified.InternetInformation (notpublished articlesor technicalpapers)Information doesnot address Table2-2key questions.These data aregenerallyacceptable forrelevant sources.Data not typicallyused for thispurpose.Data not typically used for thispurpose.Data not typicallyused for this purpose.Data not typicallyused for thispurpose.Revision No. 1 Date: October 2013 Section No. 4 Page 49 of 109

Except as noted in Section 3.0, ERG will not revi ew the quality of existing data contained in EPA databases (e.g., PCS, TRI, ICIS-NPDES) or other data bases provided by federal or state agencies but will describe known limitations of the data in reports on wh ich they are based. For existing data contained in databases provided by trade organizations (e.g., Amer ica’s Natural Gas Alliance), ERG will evaluate the source of the data and discuss with the EPA WAM the suitability of the data for the intended purpose. If ERG identifies limitations in the quality of the data from published literature or federal/state generated databases that may impact decisions regarding the overall direction of further data collection efforts or the quality of deliverables, ERG will communicate these issues to the EPA WAM. Data limitation discussions will identify where data gaps exist (e.g., in PCS, major facilities are required to submit data, but PCS data may not represent all facilities in a given geographic region because minor dischargers often are not included) and a description of data reporting requirements (e.g., TRI does not require facilities to sample and analyze waste streams to determine the quantities of toxic chemicals released, so the data included in TRI may be estimates of releases ba sed on mass balance calculations, published emission factors, site-specific emission factors, or other approaches).



### 4.3.2 Quality Review for Existing Data Used in Environmental Assessment Analysis

In addition to the analyses listed in Table 4-2, ERG will use existing data to support qualitative and quantitative assessments of the impact of industry discharges on the environment (e.g., environmental modeling to predict receiving water concentrations and fish and wildlife tissue concentrations).



### 4.3.2.1 Qualitative Analyses

Qualitative analyses provide a descriptive discussion of environmental impacts without quantifying the specific impacts. Qualitative impact analyses will rely primarily on existing data collected as part of a literature review. The criteria that ERG will use to evaluate the quality of lite rature sources used for qualitative analyses are: (1) accuracy, (2 ) reliability, and (3) representativeness.

 Accuracy . ERG assumes that the underlying data and in formation contained in state and federal reports and peer-reviewed journal articles are a ccurate. ERG assumes that data from these sources are of sufficient quality to identify potential e nvironmental impacts associated with industry discharges. ERG will present data that are outliers (e.g., data that are not consistent with the body of information collected from the literature re view) to EPA for further direction on how to incorporate or exclude the data in the analysis.

 Reliability. During the course of the literature search, ERG may encounter sources that contain limited documentation of the environmental im pact claim (e.g., unidentified speaker at a stakeholder meeting or a local resident commenti ng in a newspaper article). These data sources represent a lower level of reliability than pee r-reviewed literature, but may provide useful information for capturing the full range of envi ronmental impacts associated with industry discharges. If ERG identifies lim itations in the quality of information taken from other existing data sources (e.g., websites, ne wspapers, and stakeholder m eetings), ERG will communicate these data quality issues to EPA. ERG will also evaluate existing data for use in qualitative analyses for reliability based on the following factors:

- The scientific work is clearly written, so that all assumptions and methodologies can be identified;Revision No. 1 Date: October 2013 Section No. 4 Page 50 of 109

- The variability and uncertainty (quantitative and qualitative) in the information or in the procedures, measures, methods, or models are evaluated and characterized; and

- The assumptions and methodologies are consiste ntly applied throughout the analysis as reported in the source.

 Representativeness. ERG will evaluate existing data for use in qualitative analyses for representativeness based on whether they represent:

- The types of environmental impacts caused by wastewater discharges from the industry of interest;

- The range of concentrations observed in industry discharges; and

- The environments affected by industry disch arges. (e.g., rivers, lakes, and wetlands).

Information on environmental impacts in all environm ents may not be readily available in the published literature. When a subpopulation is selected for anal ysis (e.g., lakes and reservoirs), ERG will clearly describe the subpopulation and resulting lim itations of the analysis to EPA.



### 4.3.2.2 Quantitative Assessments

ERG may also use existing data to support quantita tive assessments of the environmental impacts of industry’s discharges through the use of environmen tal models. Input values for the environmental models may include:

 Pollutant loadings from waste streams of interest;

 Discharge flow from discharging facilities;

 Average annual receiving water stream flow;

 Partition coefficients for pollutants of interest;

 Bioaccumulation factors for pollutants of inter est for different trophic levels of fish;

 Average weight for wildlife species of interest;

 Average water ingestion rate for wildlife species of interest; and

 Food ingestion rate of prey in each fish trophic level for wildlife species of interest.

ERG will obtain input values and parameterize the en vironmental models based on data from existing data sources, including:

 Peer-reviewed literature;

 Government documents;

 Academic reports;

 Publicly available GIS datasets from government organizations (e.g., NHD Plus); and

 Databases provided by federal or state agencies (e.g., USGS stream monitoring data).

ERG will not review the quality of ex isting data contained in governme nt GIS systems or databases, but will describe known limitations of the data in re ports on which they are based. If ERG identifiesRevision No. 1 Date: October 2013 Section No. 4 Page 51 of 109

limitations in the quality of the data from published lite rature or federal/state generated GIS or databases that may impact model predictions, ERG will communi cate these issues to the EPA WAM. In addition to the quality criteria described for the qualitative envi ronmental assessment analyses, ERG will also use the following acceptance criteria to evaluate use of ex isting data for environmental modeling analyses:

 The data are from a peer-reviewed, government, or academic source;

 The geographical area of which the data are co llected are representative of the goal of the modeling analysis;

 The exposure scenarios and concentrations are either directly from relevant facilities or are representative of the types of waste streams under consideration for the work assignment or project;

 The data sets are representative and reflect th e ecological or human population of interest;

 The scientific and technical procedures, measures, methods, or models employed to generate the information are well documented and reasonable for, and consistent with, the intended application within the environmental model; and

 The data are comparable to other sources.



### 4.3.3 Data Quality Verification and Reporting

Data verification is the confirmation by examination and review of objective evidence that the data are complete, correct, consistent, and in compliance with technical requirements, es tablished standards, and contractual requirements. To accomplish this, ERG will determine at the time of collection if collected data sources and data elements meet the data quality sp ecifications set out in sections 4.1.3, 4.3.1, and 0. ERG will compare data sources and data elements to the quality specifications and document the results as part of the project file through notes in spreadsheets and/or databases, as part of project notes, memoranda, and/or documents, or when directed in specific data quality review reports. ERG will also discuss with the EPA WAM if any data or data sour ces do not meet the specifications. A review of data usability is also conducted during data analysis and ev aluation as part of the interactive process between ERG and EPA.

ERG may be directed by EPA to use data that do not meet the specifications if, for example, there are no other available data. ERG will discuss such data quality issues with the EPA WAM and obtain written approval before using these data in s ubsequent analyses. This approval will be added to the project quality file. In its final project documentation, ERG will descri be data quality reviews, including any need to use data that did not meet quality acceptance specifications.

As discussed in Section 2.8, ERG will also provide the ERG WAM with a monthly report of the status of QA activities.



### 4.3.4 Achieving the Contract’s Overall Quality Objectives

ERG’s procedures described in Section 4.1result in identif ication of a variety of existing data of interest to EPA from numerous topic-specific sources. ERG’s syst ematic review of those data against the quality objectives described in Sections 4.3.1 and 0 helps to ensure that the data are of a known and acceptable quality. Through these procedures, the existing data id entified and collected for use will be unbiased andRevision No. 1 Date: October 2013 Section No. 4 Page 52 of 109

the resulting dataset used by EPA will meet EPA’s goa l of objectivity. ERG’s procedures to ensure the security of existing data collected are described in Section 4.2. In addition, ERG’s procedures for the handling of existing data in spreadsheets and databases (presented in Sections 7.0 and 8.0 below) helps to ensure data integrity and that data are not improperly altered during the collection and/or analysis process. The objective of collecting existing data that is usab le is accomplished through ERG’s application of this section’s procedures, including methods for identifying usable sources of data, procedures for its proper maintenance and storage, and evaluation of its qua lity in consultation with the project-specific EPA WAM.Revision No. 1 Date: October 2013 Section No. 5 Page 53 of 109



### 5.0 G ENERATION OF 

### P RIMARY 

### D ATA THROUGH 

### Q UESTIONNAIRES AND 

### S URVEYS

ERG will support EPA in developing, administering, and analyzing results of electronic and/or paper questionnaires that are sent to facilities within specified point source or industry categories as part of ELG development. In addition, EPA may also direct ERG to develop and administer smaller survey efforts to collect specific primary data from a subset of entities or may direct ERG to collect primary data through a site visit. Note: A survey of nine or fewer en tities for the same information does not require an Information Collection Request (ICR).

A point source or industry category subject to a qu estionnaire effort will be defined by the EPA WAM. ERG may assist EPA in defining the sampling frame based on publicly available da ta and data collected during any preliminary or detailed study. EPA’s st atistics team is responsible for selecting sample populations, and ERG will use the statistics team’s in put to develop the questionnaire distribution list. Based on past questionnaire efforts, ERG has develope d mailout and helpline procedures to appropriately distribute questionnaires as well as data review pr ocedures and guidelines to appropriately handle responses, review, and follow-up. ERG will compile responses into a database for use in technical analyses to answer the Table 2-2 key ques tions that are presented in Section 2.2.



### 5.1 Data Gathering

This section describes data generation and acquisition via a questionnaire, su rvey, or site visit effort and covers the following four elements:

 Data collection process design;

 Data collection methods;

 Questionnaire data handling and custody; and

 Non-direct measurements.



### 5.1.1 Questionnaire Data Collection Process Design

The design of a questionnaire data collection e ffort will depend on the following factors:

Type(s) of Questionnaires Needed for a Specific Industry Category. EPA may choose to distribute screener and/or detailed questionnaires. EPA uses screen er questionnaires to quickly identify entities that are within scope of a regulatory effort and/or to id entify a subset of entities that should participate in detailed surveys to help reduce the burden of a full surv ey effort. As an example, for the airport deicing point source category, EPA conducted a screener survey of airlines and a detailed survey of both airlines and airports. The results of the airline screener survey were used to identify a su bset of airlines for the detailed questionnaire.

Types of Data Needed for a Specific Industry Category. EPA generally develops multiple questionnaire parts and sections as needed so that questions on a sim ilar topic can be grouped together. As an example, the Steam Electric questionnaire consisted of nine parts (Parts A through I) with the various parts targeting certain types of power plants or power plan t operations. Also, in addition to the engineering and other technical data, the steam electric questionnair e included a section to obtain financial and economic information for use by EPA’s financial contractor. EPA distributed the questionnaire to all coal- andRevision No. 1 Date: October 2013 Section No. 5 Page 54 of 109

A questionnaire sample frame is the list of facilities within a population that can be sampled. petroleum coke-fired power plants and a statistically sampled subset of the rest of the steam electric industry, including oil-fired, gas-fired, and nuclear power plants. Questionnaires may also be directed at specific types of entities (e.g., POTW recipients to assess the impacts of pretreatment) or specific types of information (e.g., Steam Electric industry leachate monitoring questionnaire).

Form of the Questionnaire(s). EPA has used both paper (hardcopy) and electronic questionnaires in past ELG development efforts. Because of the advantag es of direct data importing with the use of an electronic questionnaire, ERG anticipates that fu ture questionnaire efforts under Contract EP-C-12-021 will be conducted using an electronic form. However, the discussion of data gathering elements included in this PQAPP applies to both types of questionnair e formats. In addition, for small directed data collection efforts, EPA may chose to conduct tele phone surveys. In these cases, ERG will collect responses onto a data collection form developed specifically for the purpose.

Mechanisms for Handling CBI Responses in the Questionnaires. Any technical and/or financial information designated CBI in a questionnaire respon se will be handled by ERG according to ERG’s CBI security plan entitled Security Plan for Handling Confidential Business Information Under the Clean Water Act Contract EP-C-12-021 , dated November 9, 2012 and ap proved by Ahmar Siddiqui, EPA’s Document Control Officer, on January 9, 2013. EPA ’s questionnaire data collection design needs to address how CBI will be identified and sent by the survey recipient to EPA.

ERG will work with the EPA WAM, EPA statistics team , and any other relevant contractors to support the development of the questionnaire process design. EPA will lead the design development effort through a series of discussions/meetings and interim de liverables whereby each of the factors listed above are assessed for the specific industry category unde r review. ERG, EPA, and other contractors will document the resulting questionnaire collection design in a series of planning memoranda and/or documents. This documentation will include the following:

 Description of the questionnaire sample frame development;

 Summary of the systematic planning process for the questionnaire data collection; and

 Information Collection Request (ICR) Supporting Statement.



### 5.1.2 Questionnaire Data Collection Methods

EPA’s questionnaires for industry studies or ELG deve lopment collect information for answering the key questions outlined in Table 2-2 of Section 2.2. ERG supports EPA in developing the Information Collection Request (ICR) supporting statement a nd respondent burden estimates which are approved through the Office of Management and Budget (OMB) prior to a data collection from more than nine entities. ERG will also support EPAin developing the dr aft and final versions of a questionnaire. EPA solicits comment on their draft questionnaires from other EPA offices, states, environmental groups, industry, and trade associations and the questions ar e made publicly available for comment. OMB also completes a thorough review of any EPA questionnaire that is subject to the ICR process. ERG supports EPA by revising the draft questionnaire, including sp ecific questions, formatting, and response patterns, based on all comments received to make sure the qu estions capture the specific information that is required for the intended rulemaking while mini mizing overall burden to the respondents.Revision No. 1 Date: October 2013 Section No. 5 Page 55 of 109

EPA’s information collection activities are conducted un der authority of Section 308 of the Clean Water Act (Federal Water Pollution Control Act, 33 U.S. C. Section 1318). All facilities that receive a questionnaire must respond to it. ERG mails questio nnaires to those facilities identified through the sample frame development process. ERG also works w ith existing data sources a nd any applicable trade associations to review the mailing list to ensure th at facilities covered by the industry category are included and that appropriate respondent addresses/contacts are listed.

All questionnaire responses must include a certific ation statement signed by a responsible corporate official or his/her authorized representative. EPA w ill ensure compliance with the information request by using a variety of means, including, as necessary, follow-up phone calls and letters and appropriate sanctions as provided by law. ERG will support EPA, as directed, with follow-up telephone calls and letters. For electronic questionnaire files (e.g., a ques tionnaire file in Excel®), the questionnaires may be copied onto electronic media by the Government Printing Office (GPO) or at ERG (within the constraints of the contract printing clause). In these cases, the el ectronic media will be clearly labeled with the title and version of the questionnaire. For hardcopy ques tionnaires, the questionnaire may be printed by GPO or at ERG (within the constraints of the contract printing clause). In these cases, the hardcopy forms will be clearly labeled with the title and version of the questionnaire. ERG then mails the questionnaires (on electronic media or paper copy) to the intended recipients.

ERG follows specific procedures to en sure each facility receives the correct version of the questionnaire. ERG uses a mailout QA checklist (see Attachment E for an example of a mailout QA checklist) to document that the assembly of each mailout package is correct and complete. ERG develops the mailout QA checklist, customizing it as needed based on the nu mber and types of questionnaire inserts for the mailout packages. ERG also documents which pack age is sent to each respondent. ERG will create and use an industry-specific Electric Tracking Database to document questionnaire delivery using a delivery service with tracking capabilities, such as Federal Express. This database is also intended to aid ERG in keeping track of questionnaire review, discussed in Section 5.3. ERG QC activities associated with the mailout procedures include:

 Checking each printed mailing label against th e finalized mailing list and “checking off” the appropriate column on the mailout QA checklist;

 Checking 10 percent of the electronic media containing the questionnaire (or paper questionnaires) after receipt from EPA to make sure that all of the files required for the version are present and readable;

 Assembling the mailout package, including the questionnaire cover letter and enclosures, the appropriate questionnaire versi on, and mailing label, and “checking off” on the mailout QA checklist that all respondent IDs match on all mailout documents;

 Performing a 100 percent second check that the assembled package includ es the correct cover letter, enclosures, questionnaire, and return label;

 Tracking all questionnaire deliveries through Fe deral Express tracking mechanisms to confirm that the questionnaire was delivered and electroni cally transferring FedEx information into the industry Tracking Database;Revision No. 1 Date: October 2013 Section No. 5 Page 56 of 109

 Tracking all returned questionnaires to identify respondents that did not respond or that provided incomplete responses; and

 Initial follow-up via phone and written communicati ons with respondents who did not respond or respondents providing incomplete responses.

ERG will coordinate with the EPA WAM on any fu rther communication for non-responsive facilities. These communications may include a series of late le tters or follow-up calls/letters from EPA requesting completion and return of the survey or any further follow-up necessary for questionnaire review.

All questionnaire recipients will have the option to identify the information or part of the information submitted as CBI. A “CBI” checkbox is typically used ne xt to each question on the survey so that the respondent has complete control over which questi ons will be handled as CBI. See Section 5.1.3, for handling procedures of completed questi onnaires, including those marked CBI.



### 5.1.3 Questionnaire Data Handling and Custody

Completed questionnaires will either be mailed directly to ERG or transferred to ERG by EPA, where they will be logged into the industry tracking database a nd transferred (data entry or data upload) into an industry electronic questionnaire database. See Section 5.2 for a discussion of the questionnaire database. Original completed questionnaires will be logged into ERG’s Project File Index and stored as CBI in the electronic project file (for electronic responses) and in locked project file record rooms located in ERG’s offices (for hardcopy questionnaires, electronic media, hardcopy or attachments). All ERG offices are equipped with smoke alarms and sprinkler systems in case of fire. Access to the record storage areas is limited to authorized personnel and ERG institutes a signout procedure for the removal and return of any hardcopy or electronic media materials from the file record rooms.

In order to ensure data quality, ER G will not change original responses of the questionnaire during review and follow-up. ERG will create working copies to comp lete these changes. See Section 5.3.1 for further discussion of questionnaire data receipt. Any paper c opies received will be stored and used only as a reference to the original data submitted by the respondent.



### 5.1.4 Non-Direct Measurements

While preparing an industry survey effort, ERG w ill acquire, organize, and analyze existing data to characterize facilities for selection of a statistical subset of the population of interest that will receive the questionnaire, when applicable (e.g., some survey e fforts cover the full population and will not require a statistical subset). EPA may direct ERG to compile th is existing data into a sample frame database that includes all facilities potentially in scope for the questionnaire. Section 8.0 discusses ERG standard procedures for developing databases. Section 4.0 presen ts ERG’s data gathering, data management, and quality assurance procedures for existing data.



### 5.1.5 Site Visit Primary Data Collection

In addition to primary data collec tion through questionnaires and telephone surveys, ERG will also collect primary data through site visits. The types of data collected during site visits will support answering the questions in Table 2-2 and may include:Revision No. 1 Date: October 2013 Section No. 5 Page 57 of 109

 Facility specific information that characterizes the operations and wastewaters generated and discharged;

 Wastewater management and treatment operations;

 Potential locations for sampling points; and

 Facility specific information on costs, loads, and discharge pollutants, flows, and concentrations.

ERG staff will document information collected duri ng site visits in a Site Visit Report (SVR).



### 5.2 Questionnaire Data Management

The preferred format for questionnaires is an electronic information request tool (electronic questionnaire), which allows respondent s to record their responses in a computer file and submit them to EPA on an electronic medium such as a CD, DVD, or flash drive. For future questionnaire activities conducted under Contract EP-C-12-021 , ERG will work with EPA as part of the questionnaire planning and ICR process to assess the most appropriate form of the questionnaire (i.e., either an electronic form or a paper (hardcopy) form).

Upon receipt of a returned questionnaire electr onic media and/or paper survey, ERG will follow procedures outlined in the data review guidelines fo r handling the completed questionnaire, previously discussed in Section 5.1.2. ERG will conduct the re quired review of all responses, complete any necessary follow-up with facilities, and finalize any of the outstanding documentation regarding questionnaire review and follow-up.

For primary data collected as part of a site visit, ERG will document the information received in a Site Visit Report (SVR), which is reviewed by both EPA and the facility visited. Data collected during a site visit may be used to inform the sampling and anal ysis process and may also be used along with other existing data in technical analyses.



### 5.2.1 Transfer of Electronic Questionnaire Resp onses into a Questionnaire Database

ERG will transfer all questionnaire responses (CBI and non-CBI) received into one questionnaire database. The questionnaire database as a whole w ill be handled as CBI until such later time as a sanitization plan is made to create a non-CBI version for the public docket. The database will be stored electronically on ERG’s CBI server. ERG will prepare a data element dictionary to document the database structure and to serve as a guide for database users.

In order to transfer responses from MS Excel®-b ased electronic questionnaires into the questionnaire database, ERG will create a program that referenc es mapped answer cells from the base questionnaire files and transfers them into specific locations/tabl es in the database. For open-ended questions, ERG will transfer the responses into a text field of the data base. ERG will map every worksheet of the electronic questionnaire and thoroughly check all mapped refe rences in each worksheet before distribution.

ERG will perform several small “test runs” to pull diff erent parts of an electronic questionnaire into the database and will continue test runs until all data are entered into the database without error. ERG will then pull in a complete “mock-up” questionnaire to check that all mapping references and data correctlyRevision No. 1 Date: October 2013 Section No. 5 Page 58 of 109

transfer into the database to the correct table. ERG will complete a 100 percent check of the resultant test database to ensure correct data transfer.

ERG will also complete a 100 percent check of the tr ansfer of data from the first questionnaire response received for each version of the questionnaire. ER G will notify the EPA WAM of the results of the transfer. If mapping errors are detected (e.g., informat ion is populated in the wrong fields or incorrect tables), ERG will document the error, correct it, and continue to perform 100 percent checks on additional questionnaire responses until all data are transferred corr ectly. If data transfer errors are detected (e.g., data type mismatch, incorrect file name), ERG will docu ment the error, correct it, and re-extract the data from that file.

ERG developed the following aspects of the data transfer program to efficiently extract data and minimize back end work in the database.

Survey Review Table . The survey review table is a table in the database that includes a listing of all respondent IDs and the status of their complete ness check and review. The table includes several fields that are utilized to document complete and su ccessful data transfer. The first field, marked by the survey reviewer, flags the reviewed questionna ires that are ready for data extraction. After a successful data extraction, the program removes the flag that marks the questionnaire as ready and the date and time of transfer are recorded in another fi eld. However, if the data transfer is not successful, the flag that marks the questionnair e as ready is not removed, and another flag is added indicating an

error.

Error Log . The error log is a table in the database that automatically updates during data transfer. If any transfer errors occur (i.e., data type mismatch, incorrect file name), the error log will populate the respondent ID, table name, and field name where the error has occurred.

True Zero Values . Due to the nature of electronic questionnaire forms, all blank text and numeric fields are stored as “0”. ERG developed a process to distinguish between the blank data and true zero values entered by the respondent. Because both valu es are entered as zero into the database, ERG survey reviewers will change true zero responses to “-555” during technical review (i.e., before extraction of the data). ERG’s data extraction pr ogram will change any remaining blank or zero values to “-999.” After data extraction is complete , blank fields will show as “-999” and true zero fields will show as “-555.” The transfer program w ill then automatically update all “-555” values to zero. This process will represent the actual data entered into the questionnaire.

Blank Tables . Certain aspects of an electronic questionnaire inevitably result in blank responses. These include skip patterns and tables where responde nts answer only applicable rows. In order to avoid a database full of blank data, ERG’s extracti on program will be tailored to each questionnaire to skip these rows of blank data. This will resu lt in a database populated with only the information from facilities that have responded to a specific part, section, or subset of questions (depending on the skip patterns throughout the questionnaire).

To ensure that the database accu rately reflects responses, ERG will perform a 100 percent comparison of the information transferred into the database by section of the questionnaire to information recorded in the electronic survey file, for a pre-determined percent of responses. This comparison will also check that theRevision No. 1 Date: October 2013 Section No. 5 Page 59 of 109

survey review and error log tables accurately reflect the results in the database. This QA review will include checks that blank data are recorded as “- 999”, true zero responses are recorded as zero in the database, and that blank rows/tables of information not required (e.g., due to skip patterns), are not transferred into the database. This verification step will confirm that the electronic transfer operations are working correctly. Other typical database QC checks are described in Section 8.0.

Following technical review and follow-up of the ques tionnaire responses, ERG will transfer data into the questionnaire database. ERG does not expect to manually enter any data, as a result of the electronic data transfer. However, in the event any data do require manual data entry, ERG will perform a 20 percent check (or other percentage as specified by the EPA WAM) to ensure it has b een accurately transcribed from hard copy.



### 5.2.2 Transfer of Paper Questionnaire Respon ses into a Questionnaire Database

ERG will transfer all paper (hardcopy) questionna ire responses (CBI and non-CBI) received into one questionnaire database. The questionnaire database as a whole will be handled as CBI until such later time as a sanitization plan is made to create a non-CBI version for the public docket. The database will be stored electronically on ERG’s CBI server. ERG will pr epare a data element dictionary to document the database structure and to serve as a guide for database users.

Following technical review and follow-up of th e questionnaire responses, ERG will manually enter information from the paper working copy questionnai res into the database. ERG will generally use a double data entry process to identify any data entry er rors. In the double data entry process, the responses from the same questionnaire are manually entered by two different staff and a computer program compares the entered data to ensure they match. Th e double data entry check will generate an error report that will document any issues that require resoluti on. An error resolution form will be created to document any fixes that are required for these databa ses. Database revisions from this process are also documented into a questionnaire database revision/corr ection spreadsheet. As with other documentation, the error report document and revision/correction spread sheet will be submitted to EPA for approval prior to their use and the final documentation will be pl aced in the record as dir ected by the EPA WAM.



### 5.2.3 Questionnaire Data Documentation

Documentation for a questionnaire effort includes the questionnaire responses on electronic media or hardcopy, revisions or edits made by the response revi ewer in the working copy of the questionnaire, the completed questionnaire review checklist, and do cumentation of communications with respondent contacts via a helpline or follow-up contact. These documen ts, except for the original electronic media or hardcopy response, will be stored in a working file specific to each respondent on ERG’s CBI server (or non-CBI server if no CBI data are provided). The origin al electronic media or hardcopy response will be stored in the ERG project archive. The questionnaire review task leader or hi s/her designee will inspect the questionnaire review documentation to ensure th at all records are legible and complete. ERG will log these records into the project file and copies of these records will be provided to EPA as requested.

ERG will use standardized practices for database de velopment and database queries while executing the work described in this plan. Section 8.3.1 descr ibes database development quality control checks andRevision No. 1 Date: October 2013 Section No. 5 Page 60 of 109

Section 8.3.2 describes database query quality contro l checks. These practices include requirements for identification and control of differe nt versions of the database and its tables. These requirements include:

 Assigning a unique name and number to the revised version of any database tables where revisions have taken place and maintain ing a log of database table revisions.

 Assigning a unique name and number to versions of the database. Also, maintaining a log that identifies database revisions and describes the cha nges made to the differe nt versions. This log will be a table within the database.

ERG will use data in the industry questionnaire database to perform specific analyses. ERG will construct database queries to perform these analyses, follo wing the database query qua lity control procedures described in Section 8.3.2.



### 5.2.4 Management of CBI Data in a Questionnaire

Information collected through a qu estionnaire claimed as CBI by the survey respondent will be handled according to the procedures outlined in ERG’s CBI security plan entitled Security Plan for Handling Confidential Business Inform ation Under the Clean Water Act Contract EP-C-12-021 , dated November 9, 2012, and approved by Ahmar Siddiqui, EPA’s Do cument Control Officer, on January 9, 2013. Only staff demonstrating their understanding of CBI ha ndling procedures by passing a written test will be permitted to access CBI on this contract.



### 5.3 Quality Assessment

This section describes ERG’s QC procedures for prim ary data collection through a questionnaire along with the data quality criteria that ERG will assess for acceptance and use of those data in an ELG development project. ERG will assist EPA in the review of all submitted questionnaires. As described in Section 5.3.1, ERG develops Data Review Guidelines to review ques tionnaires using several methods to improve completeness, consistency, and reasona bleness of questionnaire responses. ERG typically conducts a question-by-question techni cal data review of all survey responses using the technical review checklist that defines the data acceptance criteria. ERG expects the project-specific technical review checklist to be updated periodically as the questionna ire review process evolves. Questionnaire data acceptance criteria are outlined in Section 5.3.2. ERG anticipates that the only circumstance in which a question response will be rejected would be a failure to resolve significant data issues with the respondent

contact.



### 5.3.1 Questionnaire Quality Control

Helpline . As directed by the EPA WAM, ERG will operate a questionnaire helpline to offer assistance to entities responding to a questionnaire. Dependent on EPA’s direction, questionnaire respondents will be able to call in or email specific questions rega rding the questionnaire and receive guidance on how to respond. Helpline contact information will be provide d with questionnaire instructions. ERG will operate the helpline to ensure quality and completeness of questionnaire responses. ERG will develop helpline guidelines (which are tailored to a speci fic survey) so that information pr ovided via the helpline is correct and consistent. The helpline guidelines also provide instructions for correctly documenting any helpline communication with a respondent. ERG will develop su rvey-specific helpline guidelines that include:Revision No. 1 Date: October 2013 Section No. 5 Page 61 of 109

Completeness Check

 Check that the respondent has provided all requested contact information and respondent identification (e.g., facility name and respondent ID on each worksheet of the questionnaire);

 Check that the respondent returned all applicable parts of the questionnaire;

 Check that response data submitted includes all necessary files and that all parts of the file are present and able to be viewed/accessed; and

 Check that all applicable questions are completed on each worksheet/section of the questionnaire.

 Check that the certification signature is returned and com p lete.  Requirements for determining if a respondent needs to complete certain sections of the questionnaire;

 Instructions/clarification on how to complete any part, section, question, or table;

 Checklists for all required information for bloc k/water balance diagrams (as appropriate);

 Acceptable units of measure for specific questions; and

 A log of responses to all issues encountered by respondents during questionnaire completion.

Both a telephone line and email helpline are provided fo r respondents to use to receive help in completing the questionnaire. The helpline log contains two logs, one for the telephone helpline and one for the email helpline. Any phone or email communi cations with facilities will be documen ted in the helpline log. Each phone or email message will be represented by one lin e in the helpline log. Each message will be identified by respondent ID, date, time, and assisting helpline personnel. This helpline log will assist ERG in quickly searching past questions and responses wh ile operating the helpline. As directed by the EPA WAM, the helpline log will also serve as a templa te to develop a frequently asked questions (FAQ) document that will be placed on EPA’s website and periodically updated to assist respondents while completing the questionnaire. The final FAQ document will ultimately be submitted to the public docket for the industry rulemaking.

Data Receipt . ERG has developed a procedure to ensure th e proper handling and storage of questionnaire responses. This procedure will help ensure that a ll questionnaire responses are received, tracked, saved, and copied to ERG’s CBI network. The pro cess ERG will follow upon initial receipt of any questionnaires administered under this contract includes:

 Update the Survey Tracking table in the Survey Tracking database with the date ERG received the returned questionnaire.

 Perform a completeness check using a checklist to ensure that all applicable files are returned and completed.

 Report to EPA the number of questionnaire responses that have been received to date (weekly or as directed to do so by EPA).

 Create a working folder for each respondent that has submitted a questionnaire response. Save a working copy of questionnaire responses and attachments in the working folder for questionnaire review.

 Save all data review and follow-up documentation in the working folder for each respondent.

 Periodically deliver batches of the economic andRevision No. 1 Date: October 2013 Section No. 5 Page 62 of 109

financial part of the questionnaire to EPA’s fi nancial contractor, if such sections of the questionnaires are received by ERG.

 Store the returned electronic media and any pa per documents in the ERG project archive as original files. These files will not be altered or changed, regardless of any issues flagged during data review.

Data Review . ERG will develop Data Review Guidelines to ensure the quality and completeness of the questionnaire responses. The data review guidelines in clude the data review procedure, the technical review checklist (which is unique to each questionnai re and is based on the survey questions asked), and the documentation required for questionnaire review and any resulting respondent follow-up. Attachment F presents an example Data Review Guideline from th e Airport Deicing Project. Included within the Data Review Guideline of Appendix F is an example technical review checklist.

The data review procedure and technical review checklist include both consistency checks and reasonableness checks and examples of each of these include:

Consistency Checks

 Check that information from part to part, and section to section, throughout the questionnaire is consistent (e.g., the same number of process a nd treatment units are reported throughout);

 Check that ID codes used throughout the questionna ire are not repeated or have multiple sets of information; and

 Check that the information submitted on block di agrams or flow balances correspond to the information provided in question responses.

Reasonableness Checks

 Check all applicable questions ar e within specified logic ranges;

 Check that any calculations performed by the respondent are correct; and

 Check that all information provided for one system or unit logically makes sense (i.e., the influent and effluent match for a pond system).

ERG will use the following steps when conducting questionnaire review.

1. The first technical reviewer performs a technical data review of the questionnaire response, using the technical review checklist. Completion of each check will be marked by the first technical reviewer’s initials in the technical checklist.

2. A second technical reviewer performs a QA revi ew of a specified percentage of the questionnaires to ensure the completeness and accuracy of the primary review. ERG, in consultation with the EPA WAM, will determine the percentage of QA review to conduct and will document the percentage in the Data Review Guidelines. Completion of the check will be marked by the second technical reviewer’s in itials in the technical review checklist.Revision No. 1 Date: October 2013 Section No. 5 Page 63 of 109

3. The first and second technical reviewers update the review status of the questionnaire in the Survey Review table in the industry tracking da tabase after completing questionnaire review. ERG will periodically report the numbe r of reviewed questionnaires to EPA.

4. After the first technical review is complete, fi rst reviewers will contact facilities to discuss any follow-up questions regarding responses determined to be incomplete or inconsistent. The first reviewer will also contact facilities regarding answers considered illogical or unreasonable (according to reasonable ranges identified by ERG for each question). Reasonable ranges, included in the technical review checklist, are de termined by ERG in consultation with the EPA WAM using information from industry studies, site visits, facility contact information, and/or existing datasets.

Any outstanding issues regarding the technical review checklist will be documented in the working file for each facility. Outstanding issues that require r espondent follow-up will be documented according to the data review guidelines and saved in the working file for each facility. Every telephone call and email contact with the respondent will be documented in th e Follow-up Questions tab in the technical review checklist. If a respondent requests follow-up conversati ons in writing, ERG will prepare and document all follow-up letters and responses. Once all issues are re solved, ERG will correct the working copy of the questionnaire.

During the review and follow-up process, ERG w ill make changes to the working copy of the questionnaire. The first reviewer will document every potential change to the questionnaire that results from questionnaire review and facility follow-up in the Ch ange Form tab of the t echnical checklist. This documentation will include the question number, cell reference (if EPA used an MS Excel® questionnaire), a comment explaining the change, the da te changed, and the initials of the reviewer who changed the document. During the QA review, the sec ond reviewer will confirm these changes to ensure that they are complete and appropriate. Once the sec ond review is complete, the first reviewer will make changes to the working copy of the questionnaire. ER G is making changes to the working copies of the questionnaire to minimize the number of changes made to the data once extracted into the database. All technical checklists will be saved in the working fold ers of each facility and ultimately submitted to the rulemaking record.



### 5.3.2 Questionnaire Data Acceptance Criteria

The criteria ERG will use to evaluate the quality of data collected in a questionnaire are presented in Table 5-1, along with acceptance specifications for each criterion. ERG will follow the procedures outlined in the data review guidelines for any ques tionnaire responses that do not meet the following acceptance criteria.Revision No. 1 Date: October 2013 Section No. 5 Page 64 of 109

Table 5-1. Industry Questionnaire Data Acceptance Criteria

Questionnaire Data Acceptance Criterion Description/Definition Data Acceptance Specification Unambiguous For an individual response, can the response to the question be interpreted without confusion?  Units of measure are identified.

 Responses to open-ended questions are clear. The response indicates that the question was interpreted consistently by multiple respondents. Internal consistency For an individual response, are the answers to one question consistent with answers to other questions?  Responses follow questionnaire skip patterns.

 Responses to multiple questions asking for similar or related information are comparable. Completeness (Facility Response) For a facility’s response, were all applicable questions answered? A facility’s response must include answers to all applicable questions. The EPA WAM may specify that less than complete answers are acceptable on a facility-specific basis. Completeness (Question Response) For each question, was it answered by all respondents? Each question must be answered by all respondents to which it applies. The EPA WAM may specify that less than complete answers are acceptable for specific question responses. Reasonableness For an individual response, are reported values consistent with operating procedures at the industry of interest? A facility’s response will be checked against logical responses provided in the technical data review checklist. ERG will follow up with any respondent whose responses appear to be outliers. Representativeness Did EPA receive sufficient response to the questionnaire to reflect the population of the industry of interest? EPA’s goal is to receive at least 90% response rate in each sample frame. EPA’s statistical support will determine if the number of responses received represent each sample frame.
Comparability Is the information provided in response to the questionnaire consistent with facility descriptions available from other sources, detailed study, and/or site visit data? Are unit operations reported the same as data collected from other sources, studies, or site visit data?

Because EPA’s surveys are conducted under the author ity of Section 308 of the CWA (Federal Water Pollution Control Act, 33 U.S.C. Section 1318) and response is mandatory, ERG anticipates high response rates and high completeness rates. ERG w ill determine completeness by tracking when, and how many, questionnaire responses are reported in the indust ry tracking database. In addition, ERG will report to EPA any instances of completeness less than 100 per cent and will follow up with respondents to ensure complete responses by all respondents. ERG will also document any instances of respondents not being required to complete a questionnaire due to their cu rrent operating conditions or because they are granted a response waiver by EPA.



### 5.3.3 Site Visit Primary Data Review and Acceptance Criteria

Site visit data are subject to the same data re view and acceptance criteria as survey data and the acceptance criteria displaye d in Table 5-1 apply.Revision No. 1 Date: October 2013 Section No. 5 Page 65 of 109



### 5.3.4 Achieving the Contract’s Overall Quality Objectives

By its nature, primary data collection through qu estionnaires or site visit surveys will reflect the information and view of the facility providing the data and thus will represent some facility bias. EPA and ERG work to achieve a full picture of the industr y by developing a representative sample frame for questionnaire efforts and by developing survey questions that solicit information that is factual. When “open ended” questions that may solicit opinions are re quired, ERG strives to develop a clear context for the response. ERG’s survey data handling procedures described in Sections 5. 1.3 and 5.1.4, along with ERG’s quality control procedures described in Section 5.3.1, ensure proper handling of survey data and review of those data to maintain its integrity. In add ition, these procedures also help to ensure that survey data is not improperly alte red during the collection and/or analysis process. EPA’s main goal in collecting primary data through questionnaires and site visit surveys is to identify information that is specific to an industry and/or topic of interest and thus usable for EPA’s purposes.Revision No. 1 Date: October 2013 Section No. 6 Page 66 of 109



### 6.0 G ENERATION OF 

### P RIMARY 

### D ATA 

### T HROUGH 

### S AMPLING AND 

### A NALYSIS

Under Contract EP-C-12-021, ERG may be directed by EPA to conduct sampling and analysis in support of industry studies and/or ELG development. The objectives of collecting primary sampling data are to characterize specific wastestreams, pollutants, concen trations, and flows; evaluate treatment performance of technologies and practices; and otherwise assess industry circumstances that answer the questions posed in Table 2-2 of Section 2.2.

This section describes ERG’s data gathering process us ing field sampling episodes or sampling programs, ERG procedures for management of sampling data, and QA/QC procedures that apply to sampling and analysis activities. Since sampling program design is industry- and site-dependent, ERG develops a Sampling and Analysis Plan (SAP) for each sampling epis ode to fully define the type of sampling to be conducted, matrices to be sampled, analytes to be me asured, field data to be collected, and sampling and analytical methods ERG will use to complete the sa mpling episode. Each measurement parameter in an EAD sampling episode is potentially critical; however, any measurements that are needed for information only may be noted in the SAP. Any deviations from the QA/QC procedures outlined in this section should be documented in the SAP and/or in a SQAPP as dire cted by the EPA WAM. Non-standard situations, not covered by this PQAPP, would also be documented in a SQAPP.

Generally, ERG transfers samples collected during a fi eld sampling episode or sampling program to labs specified by EPA’s Sample Control Center (SCC) . In these cases, SCC contracts with the specific laboratories and conducts review of the laboratory r esults generating data review narratives. It is also possible that as part of a wastewater sampling program, ERG may be directed by EPA to arrange for analysis of samples by appropriate analytical labor atories. In these cases, ERG will be responsible for ensuring that the laboratories analy ze samples using correct methods as sp ecified in their subcontracts and that they are following all required QA/QC procedures. ERG may also be directed by EPA to review the analytical data generated by the laboratories and prep are the data review narratives. In either case, the laboratory contracts will specify the laboratory analyses to be performed , the analytical methods, and the required laboratory turn-around-times for analytical results. ERG notes that laboratory turn-around-times are specified in each laboratory contract and may vary. EPA will pay higher costs for quicker turnaround-times which should be considered by EPA when specifying this criterion.

EPA’s Policy to Assure Competency of Laboratories, Field Sampling, and Other Organizations Generating Environmental Measurement Data under Agency-funded Acquisitions ( http://epa.gov/fem/pdfs/fem-lab-competency-policy.pdf ) requires that:

“organizations (e.g., laboratories, fiel d sampling and measur ement) generating environmental data through measurement under Agency-funded acquisitions must submit documentation of their competency... in the fields of analyses to be conducted prior to performing such analysis.”

Prior to subcontracting with or otherwise procuri ng services from an organization to perform any field sampling, field measurements, or laboratory analyses under this contract, ERG will request and evaluate appropriate documentation to verify that the organiza tion is competent to perform the applicable field or laboratory services, in accordance with EPA’s organiza tional competency policy. ERG will retain records of that evaluation as part of ERG’s files and make the evaluation records availa ble for inspection by theRevision No. 1 Date: October 2013 Section No. 6 Page 67 of 109

EPA Project Officer or designee upon request. ERG will retain the evaluation records for the duration of this contract.



### 6.1 Field Sampling

For sampling programs conducted under Contract EP-C-12-021, ERG will support EPA in designing the sampling program, developing site-specific sample plan s, collecting the samples, shipping them to the laboratories, and analyzing the sample results. ER G will collect engineering data during the sampling episodes to characterize production and flow during sa mple collection. ERG will review these data for completeness (i.e., that data are collected for each da y and each sample point) and comparability (i.e., that data compare to production and flow data reported in any plant-specific documentation).

If directed by EPA, ERG will also support EPA in designing industry self-monitoring programs, ensuring samples are collected properly, shipped to the laborat ories, arrive at the laboratories on time and in good condition, and are analyzed. ERG will also ensure that plants that are part of a self-monitoring program will provide the engineering data associated with the industry self-monitoring program samples. ERG will also review these engineering data for completeness and comparability.



### 6.1.1 Sampling Process Design

ERG will assist EPA in conducting single or multip le-day sampling episodes and obtaining monitoring data from facilities of interest to the industry st udy or ELG development pr oject. ERG will support EPA in selecting plants for sampling based on the plant’ s specific operating characteristics. The types of operating characteristics considered for selecting plants for sampling include:

 Type of industrial operation;

 Plant location, age, and/or size;

 Types of production operations and their wastewater discharges;

 Type and age of wastewater treatment and control systems;

 Types of other control operations impacting air and solid waste generation;

 Plant pollution prevention operations/procedures; and

 Viable sampling locations.

ERG will develop a site-specific SAP and/or a gene ric SAP. A generic SAP applies to a sequence of sampling episodes that are covered under one progra m and will have a similar sampling methodology and objective. The generic SAP may then be supplemented with site-specific SAP addenda or site-specific SAPs. Site-specific SAPs and generic SAPs describ e the planned sampling approach, including criteria for determining sampling point and analyte selection, sample collection procedures, specifics related to the planned analysis methods, field measurements an d engineering data collection; sample handling and shipping procedures, and contact information. At th e beginning of the generic or site-specific SAPs developed for this contract, ERG will include the ratio nale for the sampling design reflected in the SAP. ERG’s sample planning will also include a plan for pr otecting the health and safety of the sampling crew including details on location and methods to contact support services (e.g., fire and rescue, local hospital).Revision No. 1 Date: October 2013 Section No. 6 Page 68 of 109

Site-specific sampling plans and sampling plan addenda will clearly identify sampling points and corresponding list of analytes for each plant sample d. SAPs will also specify collection of process information, including process wastewater flow rat es, chemical additions, and routine monitoring data. Collection of engineering data is described in ERG’ s generic sampling plans and will be detailed in any site-specific sampling plans and addendum.

Site-specific conditions may result in modifications to the planned sampling methodology. For example, if samples collected from the influent to a Flue Gas Desulfurization wastewater treatment system (at a Steam Electric plant) are still reacting (e.g., fizzing, bubbling) after sample collection, it may be necessary to leave the bottles uncapped until th e reaction is complete. All deviations from a site-specific sampling plan will be approved by the onsite EPA represen tative at the sampling episode. ERG will document sample plan modifications (including the specifics on which samples were a ffected and information on the conditions that necessitate the modification) in both the samplers’ field notes and in site-specific Sample Episode Reports (SERs) that document th e information and data collected during sampling episodes.



### 6.1.2 Sampling Methods

ERG’s SAPs describe the planned field sampling me thodology for collecting samples from the various sample points. Each site-specific sampling plan will contain the following information:

 Name and location of site being sampled;

 General background information including, descr iption of plant operations, waste streams, and sample point and analyte selection considerations;

 Field measurement and engineering data collection;

 Specific listing of sampling points and sample matrices;

 Analytes and analytical methods;

 Sample fractions, bottle sets, and holding times;

 Field sampling QA/QC procedures and samples;

 Sampling procedures including sample collection, labeling, preservation, packing and shipping procedures (i.e., preparation of traffic reports and information on where to ship samples);

 Analytical laboratory contact information; and

 Health and safety precautio ns and support services.

ERG provides pre-sampling training to all of our sampling teams as described in Section 2.6. The training includes:

 Review of ERG’s site-specific health and saf ety requirements, including personal protective equipment and procedures;

 Site-specific health and safety training, if required by the plant;

 Sampling procedures training for all crew members, including:

- Sampling preparation;Revision No. 1 Date: October 2013 Section No. 6 Page 69 of 109

- Sample collection (techniques for collecting sampl es to be analyzed, including trace metals);

- Preservation (chemicals, ice);

- Packing and shipping, especially procedures for labeling and preparing bottles and packing coolers;

- Field tests, including temperature and pH, sulfide, aldehydes, and oxidants; and

 Sampling crew chief training (as needed) to expand on topics discussed in general sampling training.

The remainder of this section describes ERG’s procedures for obtaining, inspecting and accepting sampling supplies; obtaining, calibrating, and dec ontaminating sampling equi pment; general procedures for minimizing sample contamination; and procedur es specific to field upset conditions that impact sample collection.



### 6.1.2.1 Inspection/Acceptance Criteria for Supplies, Consumables, and Sampling Equipment

Critical supplies and consumables are those that come in contact with samples. These include sample bottles, preservation chemicals, and sampling-related c ontainers and equipment such as pickle jars, pump tubing, and scoops. Sampling crew chiefs are res ponsible for coordinating the purchase of required materials and the inspection of materials upon receipt to ensure they are usable for the specified sampling episode or program (e.g., no observable breakage, evidence of tampering or contamination). ERG will replace any materials or equipment that show evidence of tampering or contamination. Table 6-1 presents the inspection and acceptance criteria that apply to sampling supplies, consumables, and sampling equipment.

Table 6-1. Inspection and Acceptance Criteri a for Supplies, Consumables, and Sampling Equipment

Supplies, Consumables, or Sampling Equipment Inspection Procedure Acceptance Criteria Sample bottles Check condition of bottles upon receipt for breakage or tampering and collect any cleanliness certif ication forms. Sample bottles must be unbroken and certified clean by the supplier.

Preservation chemicals Check that chemicals purchased are in unopened containers and that no readily observable contamination is on the containers. Preservation chemicals must be in original packaging (unless safety requires transfer of the chemical to a smaller/clean container before use). Sampling-related materials that contact samples (tubing, scoops, pickle jars) Check the condition of any samplingrelated containers for breakage or tampering and collect cleanliness certification forms (if provided). Check sample tubing for readily observable contamination or wear. Sampling-related containers and equipment must be unbroken and clean. ERG will collect equipment blanks of sampling-related materials that contact samples to assess contamination. Non-contact sampling materials and/or equipment Check materials and sampling equipment to assess that they are in good working condition and for any evidence of tampering or contamination Sampling materials and/or equipment must be in good working condition.Revision No. 1 Date: October 2013 Section No. 6 Page 70 of 109

ERG typically generates a bottle count spreadsheet that tallies the number and types of bottles required based on the number of sample locations, days of sam pling, types of analyses, and number and type of QC volumes required. ERG then purchases pre-cleane d (and certified) sample bottles from Environmental Sampling Supply (ESS) 5 and purchases other sampling supplies and consumables from laboratory suppliers and retailers. ERG typi cally rents automatic samplers.

For each sampling episode, ERG will determine the number of sample bottles, tubing configurations, bottles of blank water, zip lock bags, trash bags, au tomatic compositors, and other supplies needed for the episode based on the length of the sampling episode or program, number of sampling points, and types of analyses to be performed. Typical field sampling e quipment that may be used for an ERG sampling program includes pH indicator paper, pH probes, ther mometers, chlorine test kits, and sampling collection devices such as peristaltic pumps, automatic samplers , and bottle dippers. ERG will order extra sampling equipment as a contingency against breakage, malfunction, or loss.

ERG will purchase certified clean sampling supplies when ever possible for anything that directly contacts samples, such as precleaned sample bottles. ERG will retain the certifications with the sampling episode documentation. For sampling supplies or equipment that contacts the sample that is not certified clean, ERG will collect an equipment blank to verify that it is not contaminated and to verify that adequate decontamination of sampling equipment was done by the sampling crew or supplier. Site-specific and generic sampling plans describe the sample collecti on techniques that will be used for collection of required blanks.



### 6.1.2.2 Sampling Equipment Calibration, Ma intenance, and Decontamination

ERG will determine the specific field sampling devices to be used for a given sampling episode based on sampling location logistics. ERG will describe th ese devices in the site-specific sampling plans. Equipment will be dedicated to each sample point (pre ferable) or the crew will decontaminate it before

reuse.

To control sampling errors, site-specific sampli ng plans will contain specifications for sampling equipment and instructions for its use (e.g., perista ltic pumps, automatic samplers , field meters, and test kits). Sampling crew chiefs are responsible for ensuri ng that all equipment is maintained in good working condition and that decontamination procedures are pr operly followed. Prior to each sampling episode, ERG will calibrate the field test equipment following the manufacturer’s instructions. All variations from the manufacturer’s instructions will be fully docume nted in the sampling field logs. Spare parts for sampling equipment will either be supplied by the manufacturer or rental company at the time of purchase/rental, or ERG will order parts as needed to maintain the equipment in working condition.

ERG may rent automatic samplers in support of E AD sampling episodes and/or programs. The rental company cleans the automatic samplers between uses using a household detergent. Prior to each rental, the rental company runs a set of diagnostic tests on the samplers and installs new internal tubing on the

5 ESS bottles are chemically cleaned using specified USEP A cleaning procedures for low level chemical analysis. Representative containers have been tested by independent certified laboratories for their appropriate use. ESS containers meet and exceed the required detection limits established by the USEPA in Specifications and Guidance for Contaminant-Free Sample Containers (OSWER Directive #9240.0-05A).Revision No. 1 Date: October 2013 Section No. 6 Page 71 of 109

sampling pump. The sampling crew will install new exte rnal tubing that will be used for the sample collection and the sampling crew will collect a field blank prior to the start of sampling during one day of the episode. Each automatic sampler will be dedicated to a specific sampling point for the duration of the sampling episode. ERG typically brings spare automatic sampler batteries into the field and spare connections and hoses as needed. If rented sampli ng equipment experiences operating problems, ERG will attempt to fix the problems or will require the rental company to supply a replacement.



### 6.1.2.3 Methods for Minimizing Sample Contamination

ERG will conduct all sampling activities in a manner th at minimizes potential contamination and crosscontamination of samples. For example, if samples are collected with a peristaltic pump, ERG will install new tubing for each sampling point at the beginni ng of the episode. ERG samplers will use noncontaminating pH indicator paper or a pH meter dur ing field tests and preservation. A separate one-liter glass jar will be filled during collection of each grab samp le set and used for field measurements to ensure that thermometers and test kits do not contaminat e the samples intended for chemical analysis. Sitespecific and generic sampling plans describe pro cedures for field measurements in more detail.

All ERG personnel who collect samples will wear new nitrile gloves at each sampling point to avoid exposure to pollutants and other chemical, physi cal, and biological hazards, and to prevent crosscontamination of samples. Samplers will use a new pa ir of gloves at each sampling point each time they collect a grab or composite sample. Samplers will u se care not to touch the insides of bottles or lids/caps during sampling. In addition, the collection of low- level trace metals requires special sample collection techniques. Site-specific and/or generic sampling plan s describe sample collection techniques that will be used for the collection of low-level metal samples.



### 6.1.2.4 Methods for Identifying Upset Conditi ons Requiring Suspension of Sample Collection

ERG’s sampling programs are intended to characterize wastewater generation and treatment during normal facility operations. Thus, if process operati ons or wastewater treatment equipment goes down during sample collection and the upset is considered within the facility’s range of normal operations, ERG will continue the sample collection. If a process upset occurs during sampling, the sampling crew chief will record the times and dates for future referenc e during the data analysis phase. A member of the sampling crew will ask facility personnel for informati on regarding the upset and record this information in the field notes. In the event of an extended upset (i.e., outside of the facility’s normal range of operations), samplers will stop sample collection. Sample collection will not recommence until normal operations resume. The ERG crew chief, in consultati on with the onsite EPA representative, will consider plant personnel’s judgment as to whether an upset is within the facility’s normal range of operations.



### 6.1.3 Sample Handling and Custody

This section describes procedures for sample handling a nd custody. Site-specific and/or generic sampling plans also contain information describing sample hand ling and custody. Each site-specific sampling plan will discuss proper sample handling in the field, including:

 Sample collection;

 Sample preservation and shipping;Revision No. 1 Date: October 2013 Section No. 6 Page 72 of 109

 Sample labeling;

 Sample set preparation; and

 Sample packing.

Sampling data that EPA uses to support regulatory de velopment programs must be legally defensible. Therefore, the sample must be tracked from sample collection through analysis. ERG will create sample labels to identify and track all samples. In addition, ERG will use field logs to document that samples were collected as described in the site-specific sampling plan.

ERG will use a traffic report to maintain a record of sample collection, sample transfer between personnel, sample shipment, and sample receipt by the laboratory. The sampling crew will complete a traffic report for each cooler or box that is shipped to a laboratory. Each traffic report will contain the following information:

 Sampling episode number for each plant sampled;

 Number of containers of each sample in shipment;

 For each sample in the shipment:

- Sample number;

- Collection date and time;

- Preservation (yes or no) and pH after preservation;

- Sample description (environmental matrix);

- Analyses required;

 Shipment or airbill number;

 Date and method of shipment; and

 The name of the person shipping the samples.

The sampling crew chief or his/her designee is r esponsible for completing traffic reports, including referencing all applicable blank and QC samples, si gning traffic reports, and noting the date and time of shipment. This individual will also inspect traffi c reports for completeness and accuracy. Any person who makes changes to the form will initia l the changes made. The sampling crew chief will retain one copy of the traffic report and two copies will accompany the sample shipment.

If ERG does not anticipate shipping samples collected as dangerous goods, the samples will be shipped under the exclusion allowed for transporting labor atory samples in 40 CFR 261.4(d) and under the definition and exception in CFR 49 Part 173.134. For any samples that do not qualify for this exclusion, the field sampling crew will package and label sampl es for shipment in complia nce with current U.S. Department of Transportation and International Air Transport Associ ation dangerous goo ds regulations.

After ERG ships samples to the designated analytical laboratories, ERG will follow up with either SCC or the laboratories (if ERG conducted the laboratory contracting) to check on sample receipt. Any sample loss (due to bottle breakage) will be identified by th e lab and ERG will work with the laboratory and the EPA WAM to determine a corrective action. Typical corrective actions for bottle breakage include laboratory analysis using extra sample volume or ERG may be able to collect additional sample forRevision No. 1 Date: October 2013 Section No. 6 Page 73 of 109

analysis to use in place of the lost sample(s). ERG wi ll document any missing or lost sample results in the SER.



### 6.1.4 Field Data Management

Field documentation tools for sampling under Contr act EP-C-12-021 include sample labels, field logs, traffic reports, and flow and process data recorded on episode-specific data collection sheets completed by the samplers. The sampling crew chief is responsib le for creating any episode-specific data collection sheets and reviews all field documentation during the sampling episode to ensure that all records are legible and complete. During the sampling episode or shortly thereafter, EPA will request any needed operating data from the plant engineers and will reque st that it be provided to ERG. ERG will confirm that all requested operating data are collected e ither during the sampling episode or received via mail/email. As needed, ERG will transfer or import fiel d operating data into MS Excel® tables for use in the SER or placement in the sampling episode file that becomes part of the project file.



### 6.1.4.1 Direct Measurements

ERG will maintain a field log notebook and/or log sh eets to document direct measurements taken at the time of sample collection. These direct measurements may include:

 Sampling location and corresponding aliquot or sample collection date and time;

 Sampling location conditions (e.g., weather conditi ons if outdoor sample, sample wastewater temperature and/or pH);

 Sample preservation chemicals and amounts added per sample; and

 Field measurements that may impact laboratory preparation procedures (e.g., field measurement of chlorine content).

The site-specific SAP will specify direct measurem ents required for each sampling episode. ERG will use this information to provide the analytical laboratory with sample conditions that may impact the analysis and as supplemental information to help understand sampling circumstances if analysis results are unusual or unexpected. The crew ch ief will be responsible for ensuring that the sampling team uses and maintains the field log notebook and/or log sheets. At the completion of the sampling episode, the field log notebook and/or log sheets are stored and maintained with the project files.



### 6.1.4.2 Non-Direct Measurements

EPA’s sampling programs are usually augmented a nd complemented by the following types of data generated and provided by the plants sampled:

 Process flow diagrams;

 Daily process operation data;

 Daily wastewater flow rates through the treatment systems;

 Treatment chemical rates of use (e.g., L/min); and

 Sludge generation (tons/day).Revision No. 1 Date: October 2013 Section No. 6 Page 74 of 109

ERG will also obtain additional relevant information on treatment techniques specific to the particular waste stream being sampled, such as retention time through the system and design and operating characteristics. For example, for sampling episodes at steam electric power plants, ERG/EPA gathered information such as coal usage, plant capacity, w astewater flow rates, sludge generation rates (if applicable), and retention times in the wastewater treatment process.

ERG will develop site-specific data sheets for engi neering data to be collected during the sampling episode. These data sheets will be included in the site-specific SAP and completed for each facility sampled. This information will be used to determine if the specific design or operational criteria of the facility’s process operations affect the wastewater char acteristics. The crew chief will be responsible for using the data sheets and ensuring that the specified data are collected. At the completion of the sampling episode, any non-direct data collected is stored and maintained with the project files.



### 6.1.5 Field Quality Control Measures and Goals

Sampling QC activities are designed to measure or estim ate the effect of sampling error such as sample contamination, effectiveness of decontamination pr ocedures, and imprecision associated with sample collection. Site-specific sampling plans will specify qu antities and types of blank and duplicate samples to be collected during each episode and ERG will work with the analysis laboratories and/or SCC (if applicable) to specify the additional sample volume to be collected for laboratory QC samples. The following subsections describe types of blank and duplicate samples that ERG’s sampling crews will collect during a sampling episode to assess field sampling QC.

Field QC samples include trip blanks, bottle blanks , field blanks, equipment blanks, and field duplicate samples. Table 6-2 presents a list of these field QC samples, example analytes, purpose, and example frequency of collection. Note that not all analytical methods require all of the QC measures listed. Sitespecific sampling plans will discuss collection of th ese samples in greater detail. The goal for sampling blanks is non-detected values for each blank sample . If detected values are found in blank samples, ERG will work with the EPA WAM and analysis laboratory to determine the likely cause of the contamination and the impact of the result on the usability of th e sample analyses from the sampling episode. The goal for field duplicates is discussed in Section 6.1.5.5.

Table 6-2. Field Quality Control Samples

Field Quality Control Example Analytes Purpose Example Frequency Trip Blank Volatile organics To monitor organic contamination of samples during transport, field handling, and storage One trip blank per episode for specified analytes.

Bottle Blank Total and dissolved metals, low-level total and dissolved metals, organics To monitor pre-cleaned bottles for metals and/or organic contamination For low-level metals, one for each method at each sampling epis ode. For routine metals and organics, one for each method for the sampling program.Revision No. 1 Date: October 2013 Section No. 6 Page 75 of 109

Table 6-2. Field Quality Control Samples

Field Quality Control Example Analytes Purpose Example Frequency

Field Blank Total and dissolved metals, low-level total and dissolved metals, organics To monitor potential contamination from sampling techniques (equipment used or ambient conditions)  For routine total and dissolved metals and organics, one blank will be collected for each type of sampling technique or special equipment used. At a minimum, one blank per episode will be collected.Field blanks will be collected for lowlevel total and dissolved metals at each sampling location and for each method.

 For hexavalent chromium, one blank will be collected for each sample point.

 For mercury, one blank will be collected for each sample point.

Equipment Blank Low-level total and dissolved metals, organics To document adequate decontamination of sampling equipment before use by the sampling crew Once per lot of equipment cleaned (for each method).

Field Duplicates All analytes except HEM/SGT-HEM To evaluate total measurement precision and assess potential sources of data variability, including sample collection, handling, preparation, and analysis One duplicate sample for every ten samples collected at a sampling event, excluding field blank samples.



### 6.1.5.1 Trip Blank

If required, the sampling crew will prepare a trip bl ank in ERG’s sample room before the start of the sampling episode by pouring an analyte-free matrix (A STM Type 1 water) into one set of sample bottles. ERG will preserve the samples and transport them unopene d to the facility. Trip blanks are stored at the sampling staging area throughout the sampling episode and then placed on ice during the last day of sampling and shipped unopened to the laboratory. Laboratories typically analyze these samples for volatile organic compounds. The goal is non- detected values for each trip blank.

Note: EPA may elect to have a trip blank tested for other analytes (e.g., dioxins and PCBs). If EPA requests analysis for other analytes, appropriate sample bottles and volumes must be identified and specified in the site-specific SAP.



### 6.1.5.2 Bottle Blank

If required, the sampling crew will prepare a bottle bla nk in ERG’s sample room before the start of the sampling episode. The bottle blank samples are an an alyte-free matrix (ASTM Type 1 water) poured into the appropriate sample bottles (e.g., 250-mL plastic bo ttle for total metals) and stored for several days prior to shipment to the laboratory. Laboratories t ypically analyze these samples for metal and/or organic compounds. The goal is non-detected values for any bottle blanks analyzed.Revision No. 1 Date: October 2013 Section No. 6 Page 76 of 109



### 6.1.5.3 Field Blank

The sampling crew will collect equipment field blanks for all sampling techniques used by sampling staff and for special equipment used during the sample collec tion, such as cooling coils, tubing, etc. The field blank evaluates potential contamination of the samp le from either the equipment used or ambient conditions. To collect field blanks, ERG will pour or pump ASTM Type 1 wate r through all equipment used for the sample collection at the sampling locatio n and into sample bottles. Multiple field blanks may be required if different sampling techniques are used or if ambient conditions are expected to vary from sample point to sample point. ERG will assist EPA in selecting analyses for the field blank using requirements of the analytical methods. Laboratori es will analyze the field blank for the specified analytes. Site-specific sampling plans will list field bl anks to be collected at each site. The goal is nondetected values for each field blank.



### 6.1.5.4 Equipment Blank

If non-dedicated sampling equipment (that contacts the sa mples) is used for mult iple sampling episodes or between sampling points of a sampling episode, ERG will perform decontamination of that equipment before its next use. ERG will collect equipment bl anks to evaluate that adequate decontamination occurred. To collect equipment blanks, ERG will pour or pump ASTM Type 1 water through/over the decontaminated equipment and into sample bottles. Multiple equipment blanks may be required if the equipment is decontaminated multiple times. ERG wi ll assist EPA in selecting analyses for the equipment blank(s) using requirements of the analytical met hods. Laboratories will analyze equipment blank(s) for the specified analytes. Site-specific sampling plans will list equipment blanks to be collected at each site. The goal is non-detected values for each equipment blank.



### 6.1.5.5 Field Duplicates

Each site-specific sampling plan will specify sample points for collection of field duplicates. ERG will assist EPA in selecting duplicate points based on inform ation from each site and the goals of the project. EPA will collect one duplicate sample for every ten samp les collected at a sampling event, excluding field blank samples. For example, EPA anticipates collecti ng four samples from each sampling point selected at a plant; therefore, if EPA collects samples from two sample points at a plant (total of 8 samples), only one duplicate sample will be collected. However, if EPA collects samples from three sample points at a plant (total of 12 samples), then two duplicate samples will be collected. The field blank samples associated with the sampling points will not be incl uded in the count of th e total samples collected.

Field duplicate samples are collected to assess the degr ee to which repeated measurements give the same results and reflect both sampling and analysis precisi on. ERG will calculate the relative percent difference (RPD) between field duplicate results. An RPD less than or equal to 50 percent is usually considered acceptable; however, RPD values greater than 50 pe rcent may also be acceptable by EPA depending on the purpose and intended use of the data. ERG will work with the EPA WAM to determine on a sampling episode/program basis what RPD values are acceptabl e to meet EPA’s quality goals and this acceptance criteria will be documented in the site-specific SAP.Revision No. 1 Date: October 2013 Section No. 6 Page 77 of 109



### 6.1.6 Sampling Program Data Qua lity Criteria and Goals

The goal of EAD’s sampling programs is to gather sampling data of the quality necessary to support EPA in evaluating industry studies and developing ELGs fo r EPA-specified point source categories. ERG will evaluate the quality of collected data using the criteri a described below. Factors that ERG will consider in reviewing data quality include completeness, sensitivity, pr ecision, representativeness, sample contamination, and comparability. The field quality c ontrol measures described in Section 6.1.5 apply to the precision and sample contamination quality criteria. Analytical quality control measures described in Section 6.3.4 apply to the precision, sensitivity, an d sample contamination quality criteria. Table 6-3 presents these data quality criteria, ERG’s sampling t echniques to control them, the method of measuring the criteria, and desired goals for the measureme nt of the criteria for EAD sampling episodes and programs. ERG notes that Table 6-3 presents data quality goals; however, ERG’s discussion of sampling results with the EPA WAM may result in a determina tion that data are acceptable for use even in cases where the measurement goal is not met. ERG will doc ument findings of the field QC review and will describe limitations of the data, if any, in the SER s or other project documentation directed by the EPA WAM prepared for each sampling episode. In general, data quality issues are identified and corrected via frequent communications between ERG, the EPA WAM, and the analytical laboratories.

Table 6-3. Field Sampling and Analytical Data Quality Review Criteria

Data Quality Review Criterion: Sampling Description/ Definition Sampling Techniques to Control this Quality Criterion How this Quality Criterion will be Measured Goal for this Measurement
Completeness The degree to which the planned sampling program is accomplished.  Use method-specified preservation and type of bottles.

 Ship samples to meet sample hold times.

 Pack and ship samples to minimize or eliminate breakage.

 Collect sufficient sample volume to account for all planned sample analyses.

 Sample collection, shipping and storage meets method-specified temperature requirements. Percent of planned samples reaching the analysis laboratory and usable for analysis. Sufficient sample volume is shipped and received by the laboratory in sufficient time to analyze all planned sample points.

Sensitivity The capability of a method or instrument to discriminate between small difference in analyte concentrations..  Choice of method – some methods for the same pollutant have better sensitivity.

 Collect method-specified volume (insufficient volume can result in high detection limits). Laboratory reported quantitation limits. Meet method specifications.Revision No. 1 Date: October 2013 Section No. 6 Page 78 of 109

Table 6-3. Field Sampling and Analytical Data Quality Review Criteria

Data Quality Review Criterion: Sampling Description/ Definition Sampling Techniques to Control this Quality Criterion How this Quality Criterion will be Measured Goal for this Measurement

Precision (Reproducibility) Degree to which repeated samples and measurements give the same results. Represents both sampling and analysis precision. Use standard operating procedures for both field sampling procedures and laboratory analysis. Collect field duplicates of not less than 10% of study samples and calculate relative percent difference (RPD). See Section 1.1.5.5 Laboratory analyzes ongoing precision and recovery standards. See Section 6.3.5.2 RPD ≤ 50% or as determined on a sampling episode/program specific basis. OPR meets method specifications.

Representativeness Degree to which sample has the same properties as other samples from other waste streams of a similar origin.  Select sampling points that will result in the most representative sample possible. Use sample techniques that promote the collection of representative samples based on sample point selection. Analytes detected and concentrations are comparable to those from other waste streams of a similar origin. The analytes and concentrations are consistent with engineering judgment of the
waste stream origin.

Sample Contamination Unintentional introduction of material into the sample during collection, transport, and/or analysis.  New sample tubing at each sampling point.

 Use of clean gloves at each sampling point.

 Conformance to clean hands/dirty hands collection methods, where
necessary.

 Use standard analytical methods, conducted using standard procedures. Blank samples (trip blank, bottle blank, field blank, and equipment blanks) will be used to

measure contamination from sampling. See Sections 1.1.5.1 through 1.1.5.4 Laboratory blanks will be used to

measure contamination from analysis. See Section 1.3.5.4  No target analytes detectable in the blank samples (i.e., above the method detection limit).

 Determinationthat any nontarget analytes detected in the blank sample will not impact use of the sampling results data.Revision No. 1 Date: October 2013 Section No. 6 Page 79 of 109

Table 6-3. Field Sampling and Analytical Data Quality Review Criteria

Data Quality Review Criterion: Sampling Description/ Definition Sampling Techniques to Control this Quality Criterion How this Quality Criterion will be Measured Goal for this Measurement
Comparability High comparability means that data can be combined for decision making
purposes.  Use standard criteria for selecting sampling points and collecting samples.

 Use standard analytical methods, conducted using standard procedures.

 Compare analytical methods and the detection limits they achieve to select the most appropriate methods. Compare waste characteristics determined through sampling between two points (either within a site or between sites)  Data compared represent the same type of
wastestream.

 Sampling points are equivalent.



### 6.1.6.1 Completeness

Completeness is defined in terms of percent of samples planned to be collected that are collected, and which the laboratory determines are acceptable for analysis. ERG will document sampling completeness in the SERs.



### 6.1.6.2 Sensitivity

Sensitivity generally refers to the capability of a met hod or analytical instrument to discriminate between small differences in analyte concentration. ERG w ill document laboratory quantitation limits (QLs) for non-detected sample results in its SERs. Laborat ory QLs may be higher or lower than the method detection limit and understanding the QLs that appl y to the reported data will inform EPA’s decision process on the use of the data.



### 6.1.6.3 Precision

Precision is a measure of agreement among repeated measurements. To evaluate precision, ERG will calculate the RPD among results from the field duplicates using the following equation.



##   100
D S 2 1 D

- S RPD 

 

where: 

### S = First sample result; and D = Duplicate sample result.Revision No. 1 Date: October 2013 Section No. 6 Page 80 of 109

ERG will evaluate the relative percent differen ce and document this evaluation in the SER. ERG anticipates collecting one field duplicate sample for every 10 samples collected per episode

. The results of the RPD analysis and the context will be examined to determine the precision of the data.



### 6.1.6.4 Representativeness

ERG’s sampling efforts measure pollutant characteris tics of selected wastewaters. Thus, the sampling points that EPA and ERG select for each sampli ng episode are designed to represent particular wastewaters and evaluate candidate treatment tec hnologies. In doing so, EPA must target a limited number of facilities having the desired characteris tics. Section 6.1.1 discusses ERG’s sampling process design.



### 6.1.6.5 Sample Contamination

Sample contamination may cause a difference between measured value and true value. Blank samples (e.g., equipment blanks and bottle blanks) are collect ed and analyzed to estimate if there is sample contamination. ERG’s goal is that no target analytes are detected in these sampling blanks. Any sample contamination is documented and ERG will discuss the impact of any detected contamination on interpretation of sample measurements in ERG’s SERs.



### 6.1.6.6 Comparability

Comparability is a qualitative measure of the confiden ce with which datasets can be compared. Sampling programs conducted under this contr act will achieve data comparability by following approved, standard field sampling procedures and by reporting results in standard units of measure required in the methods (analytes measured, procedures used, and detection li mits achieved). In addition, the equivalence of sampling locations and operating characteristics will be evaluated.

For those sampling programs that include industry self-monitoring data, ERG will identify specifics related to those data submitted (e .g., information on the methods repr esented by the data, sampling points, and conditions specific to the time of sampling). ER G will review data generated from the EPA sampling program and data generated from the industry self-mon itoring program to evaluate the comparability of the data for each sampling point over the course of the sample collection time period. Specifically, and as directed by the EPA WAM, ERG will calculate the m ean pollutant concentrations for pollutants of concern and evaluate whether the data collected from the self-monitoring program are significantly different from the EPA sampling program data using a t-test or other statistical test at a level of significance to be determined in consultation w ith the EPA WAM and EAD's project statistician.



### 6.2 Requirements for Laboratories Part icipating in EAD Sampling Programs

If ERG will subcontract directly with analytical la boratories supporting a specific sampling episode or program, ERG will ensure that th e following requirements are met:

 The laboratory will be asked what certifications or accreditations they hold and for which matrices and/or methods.

 The laboratory must document that they have r un the required methods and can meet the method specified QC requirements.Revision No. 1 Date: October 2013 Section No. 6 Page 81 of 109

 The laboratory must have a comprehensive labor atory QA program in place and operating at all times during the performance of tasks supporting the st udy. The laboratory must adhere closely to all QA/QC requirements of any SQAPP or a project-specific QAPP. The laboratory may perform additional QC to assure the validity and reliability of results and a description of any additional QC must be provided in the data package submitted to ERG.

 The laboratory must have a comprehensive data management plan that is consistent with principles set forth in EPA 2185 – Good Automated Laboratory Practices
, EPA Office of Administration and Resources Management, August 10, 1995. This data management plan will be in place and in use at all times during the performance of tasks under this contract.

 The laboratory must maintain internal records in an organized manner to document adherence to the ERG subcontract, and these records must be available for review upon request.

 The laboratory must perform all necessary QA/QC requirements specified in the analytical methods.

 The laboratory shall perform MDL studies for all analytes based on the analytical method that will be used for the sampling program. Proce dures and calculations for performing MDL studies are specified in 40 CFR Part 136, Appendix B. The laboratory must also determine a quantitation limit (QL) or “minimum level” for all analytes. Th e QLs are the lowest concentration that can be reliably measured within specified limits of precision and accuracy for a specific laboratory analytical method during routine laboratory operatin g conditions. The QL is the student t- factor times the MDL. For the seven replicates typically used in estimating MDLs, QL = 3.14 x MDL.

 The laboratory must perform MS/MSD/laboratory duplicate analyses on samples specified by ERG and samples from EPA’s sampling program must be run in the same batch as these samples. Additionally, the laboratory must spike sampl es within the spiking range specified by the analytical method or SOP. If an MS, MSD, or laboratory duplicate fails to meet the method specified criteria, the laboratory must communicat e the failure to ERG and work on resolving the issue (e.g., attempt to rerun the analysis with a higher dilution).



### 6.3 Laboratory Analysis

Field sampling episodes and/or sampling programs conducted under Contract EP-C-12-021 will include ERG or facility shipment of samples to SCC or ERG subcontracted laboratories for analysis. The subsections below describe sample handling and cu stody procedures, laboratory instrument/equipment calibration requirements, identification of anal ytical methods, typical analytical data QC samples/procedures, analytical data management, and analytical data reporting. The responsibility for any required laboratory supplies and c onsumables will rest with the laboratories contracted to perform analysis.

The goal of these procedures is to support the inte grity of EPA’s primary data collection through sampling and analysis. ERG will specify any modifi cations or deviations from these procedures in a SQAPP specific to the project.Revision No. 1 Date: October 2013 Section No. 6 Page 82 of 109



### 6.3.1 Laboratory Sample Handling and Custody

When the laboratory recei ves samples, the laboratory sample cust odian will check all bottles against the Traffic Reports, record the condition of the samples, and sign, date, and mark the time of receipt on the Traffic Reports. The laboratory sample custodian will immediately report any discrepancies or problems to ERG. ERG will verify the receipt and condition of all samples, including temperature and pH, if required by the method. Samples must be stored according to the method requirements, as well as to prevent degradation and contamination.

The laboratory must have an SOP for sample custody within the laboratory. The laboratory will log all samples into a sample rece ipt logbook or computerized laboratory information system and will document the following information:

 Date and time of sample receipt;

 Project number and name;

 Field sample number;

 Laboratory sample number;

 Sample matrix;

 Analytical parameters;

 Storage location; and

 Log-in person’s initials.

Typically samples are identified by a numbering system to mask location information and would not be handled as CBI. If, however, samples and analysis resu lts are being held as CBI, all information relevant to the samples will be secured at the end of each business day and all samples will be stored in a designated storage refrigerator with restricted access.

ERG will use an electronic sample tracking system to track the last date sa mples associated with each episode were received at the laboratory and the date of receipt of analytical data from the laboratories. The laboratory is responsible for ensuring that samp les are prepared and analyzed within the required holding time specified by the method.

If directed by EPA, ERG will subcontract analysis of EPA- and industry-collected samples to analytical laboratories. In some cases, EPA might identify analys is laboratories and ERG will ship samples to those locations. Following laboratory analysis of samples, ERG will, as directed, perform a review of the analytical data using method-specific review checklists to ensure the data are complete and reasonable.



### 6.3.2 Analytical Methods

Site-specific and generic sampling plans identify th e analytes that will be measured for a sampling episode or program and the anticipated analytical methods that the laboratories will use in sample analysis. ERG will work with the EPA WAM and S CC (if appropriate) while planning any sampling episodes or programs conducted under this contract to determine which analytical methods should be specified. ERG and EPA’s determination of the analyti cal methods will consider the type of wastestreams sampled, anticipated pollutant concentrations and the potential for matrix interferences, the desired detection and quantitation limits, and whether th e methods are EPA-approved (e.g., 40 CFR Part 136 methods).Revision No. 1 Date: October 2013 Section No. 6 Page 83 of 109

For situations where ERG is coordinating laboratory analysis and EPA has chosen not to specify the analysis method(s), laboratories may be allowed to id entify their preferred method for analysis of certain analytes and the quality control limits that would appl y at the time they submit their bid response package to ERG. In these instances, ERG will work with th e EPA WAM and the laboratory to review and approve the use of the preferred method and the analysis-speci fic QC limits. These QC limits may be set by the method, from a consensus organization compendium , or based on limits from similar EPA methods. Either way, the site-specific sampling plans will documen t the specific list of analytical methods that will be used for that sampling episode or program. Site -specific sampling plans will also identify the sample volume to be collected, the QC sample volume to be collected, the sample container type, and typical preservation required for each parameter.



### 6.3.3 Instrument/Equipment Calibration and Frequency

If ERG will subcontract directly with analytical la boratories supporting a specific sampling episode or program, ERG will confirm that laboratories selected to analyze samples are in compliance with the method required instrument calibrations and calibrati on frequencies. Instrument calibration procedures, including frequency specifications, tune performan ce checks, and maintenance requirements, are specified in the analytical methods. ERG’s PQAPP or project-specific QAPP will be provided to the laboratory upon award of the subcontract and the la boratory will be required to include a quality assurance statement in the data package submitted to ERG which verifies that the analytical method specifications have been met. In addition, the laborat ory must maintain internal records in an organized manner to document adherence to the ERG subcontr act, and ERG may review these documents as needed including documentation of instrument/equipment calib ration checks. Whenever changes are made to a system or method, the laboratory must ensure that its equipment is operating properly by performing the appropriate QC tests. Analytical instruments requiri ng calibration must be uniquely identified to allow proper association of calibration records. To ensure th at the calibration criteria are met, the laboratory must analyze test standards that are certified for purit y, concentration, and authenticity or prepared from materials of known purity and composition. Laborator ies must also perform preventive maintenance on all of their analytical equipment. Equipment mainte nance log books must contain records of equipment usage, maintenance, modificati ons, calibrations, and repairs.



### 6.3.4 Analytical Quality Control

The following subsections describe the analytical QC measures that EAD typically uses for chemical analyses conducted under their sampling programs. Note that not all QC measures listed are required for all methods and the following sections are not all in clusive. Laboratories must follow the QC procedures required by each method. The recommended frequencies identified in subsections 6.3.4.2 through 6.3.4.5 will apply unless otherwise specified in the SA P, an SQAPP, or a project-specific QAPP.



### 6.3.4.1 Method Detection Limits (MDLs)

An MDL is defined as the minimum concentration of a substance that can be measured and reported with 99% confidence that the analyte concentration is gr eater than zero. For each method, laboratories are required to perform MDL studies for each instrument on which analyses will be performed and any time a modification is made that will affect the MDL. La boratories must demonstrate that they obtain a MDL value that is less than or equal to (1) the MDL (i f listed in the method), or (2) one-third the minimumRevision No. 1 Date: October 2013 Section No. 6 Page 84 of 109

level (ML, if listed in the method). Appendix B of 40 CFR Part 136 describes the EPA procedure used for establishing MDLs. This procedure requires the laboratory to analyze at least seven aliquots of a standard spiked at three to five times an estimated MDL. Th e laboratory processes the aliquots through all steps of the analytical method and calculates the MDL as approxi mately three times the standard deviation of the mean for the analyses. The number of aliquots anal yzed determines the mu ltiplier of the standard deviation.



### 6.3.4.2 Ongoing Precision and Recovery (OPR) Standard

The laboratory will prepare an ongoing precision a nd recovery (OPR) sample (such as an aqueous performance standard for metals analysis) by adding known amounts of the analytes of interest to a matrix with known characteristics. Reagent water will usually be used as the reference matrix. The standard will be processed through all sample preparation and analys is steps. Analyte recoveries are evaluated against method-specific limits. Note: The sta ndard used should be from a source different than that used for the initial calibration standards. If a sec ond source standard is not available, the standard should be from a different lot.

Purpose: To assess laboratory perform ance and evaluate error sources in the laboratory.

Recommended Frequency: As specified by the method or subcontract.

QC Acceptance Criteria: As specifi ed by the method or subcontract.



### 6.3.4.3 Calibration Check Standard

The laboratory will perform a calibration check standard by adding analytes of interest to a laboratory reagent blank. Analyte recoveries are evaluated us ing limits specified in the applicable method.

Purpose: To ensure that the instru ment calibration remains in control.

Recommended Frequency: Every 10 analytical runs, or every 8 to 24 hours, as specified in the method.

QC Acceptance Criteria: As specifi ed by the method or subcontract.



### 6.3.4.4 Laboratory Blanks

The laboratory will analyze a laboratory blank, which c onsists of reagent water or “clean matrix” material that does not contain the analytes of interest or interfering compounds. ERG requires the laboratory to analyze a laboratory blank so that any sample c ontamination originating at the laboratory can be identified.

Purpose: To ensure that the analys is system is free from interferences and contamination.Revision No. 1 Date: October 2013 Section No. 6 Page 85 of 109

Recommended Frequency: Per method and subcon tract requirements, which specify a per batch preparation frequency or an 8- to 12-hour per analysis frequency.

QC Acceptance Criteria: As specifi ed by the method or subcontract.



### 6.3.4.5 Matrix Spike/Matrix Spike Duplicate/Laboratory Duplicate

Depending on the requirements of the analytical met hod, the laboratory will usually perform a Matrix Spike (MS), and a Matrix Spike Duplicate (MSD)/or a Laboratory Duplicate. A MS is a sample prepared by adding a known mass of target analyte to a specified amount of matrix sample for which an independent estimate of target analyte concentration is available. Spiked samp les are used to determine the effect of the matrix on a method’s recovery efficiency.

Laboratories prepare an MS by adding known amounts of compounds of interest to the field sample. The spiked samples receive the same ha ndling, extraction, cleanup, and analysis steps as the field samples. The laboratories then perform a percent recovery (%R) calculation:



### 100 SA ) SR SSR ( %R  



where: 

### SSR = Spiked sample result; SR = Sample result; and SA = Spike added.

If possible, the laboratory should determine the approp riate spiking concentration for the MS sample by first analyzing the original sample. ERG notes that this practice is not always conducted. In general, the spiking concentration should be at least twice the concentration of the analyte present in the original sample and ideally should be no more than 10 times that concentration.

An MSD or a laboratory duplicate is a repeat analysis of a MS sample or an internal lab QC sample prepared and handled in a duplicate manner to the origin al sample and is used to evaluate analytical or measurement precision but not the precision of sampling, preservation, or storage internal to the laboratory. In general, organic methods will re quire an MS/MSD, inorganic methods will require a MS/laboratory duplicate, and some methods may require neither. EPA, in consultation with the laboratory and ERG, will review the method-specified QC sample analysis and volumes and will determine whether additional QC analysis is desired. ERG will documen t the laboratory QC requirements in the contracts with specific laboratories and QC sample volumes in the SAP. Typically, ERG will collect one QC sample volume (i.e., MS/MSD or MS/laboratory dupli cate sample) for every ten samples collected at a sampling event, excluding field blank samples. Note specific to metals: If the MS or MSD analysis cannot be met using procedures described in the method (e.g., dilutions), the laboratory must perform a post-digestion spike on the parent sample after digest ion to verify that the MS/MSD failure is due to matrix interferences.Revision No. 1 Date: October 2013 Section No. 6 Page 86 of 109

Purpose: MS: To determine the e ffect of the matrix on a method’s recovery efficiency.

Duplicate analysis: To evaluate analytical or measurement precision.

Recommended Frequency: One MS/MSD or MS/laboratory duplicate sample for every 10 samples collected at a site.

QC Acceptance Criteria: As specifi ed by the method or subcontract.



### 6.3.5 Preparation and Analytical Batches

The laboratory must perform preparation and anal ysis in batches as defined by the method. The appropriate QC samples must accompany the batches as specified in the method. A preparation batch consists of all samples prepared/extracted/digested together by the same person during the same work day. An analytical batch consists of samples analyzed using the same instrument within 24 hours, or a method specified time period. Typically, batches c ontain no more than 20 field samples (i.e., non-QC samples).



### 6.3.6 Internal Standards, Labeled Compounds, Surrogates

Internal standards may be required by the method fo r calibration and quantitation purposes. They are used to measure the relative responses of other method analyt es and surrogate standards. Internal standards are added to the concentrated sample extract prior to anal ysis. When required, the internal standards must be added to all samples and QC samples at the same con centration. The recovery of the internal standard must meet the method specified criteria.

Labeled compounds may be required by the method for calibration and quantitation purposes. The labeled compounds are used to correct for the recovery of the corresponding analytes. Labeled compounds are added to the sample prior to extraction. When re quired, the labeled compounds must be added to all samples and QC samples at the same concentration. The recovery of the labeled compounds must meet the method specified criteria.

Surrogate analytes may be required by the method. They must be added to the sample before the sample is extracted. The purpose of the surrogate is to monitor method performance from extraction through analysis. When required, the surrogates must be added to all samples and QC samples at the same concentration. The recovery of the surrogate must meet the method specified criteria.



### 6.3.7 Non-Standard EPA Wastewater Methods

If non-standard EPA wastewater methods (i.e., those not included in 40 CFR Part 136 or those lacking QA/QC procedures) are anticipated for an EAD samp ling episode or program, ERG will work with the EPA WAM and participating laboratories to determine appropriate analytical QC requirements which will then be specified in an SQAPP or a project-sp ecific QAPP. Per 40 CFR Part 136.7, EPA’s guidance on QA/QC requirements for methods that lack QA/QC procedures includes three options. First the laboratoryRevision No. 1 Date: October 2013 Section No. 6 Page 87 of 109

may refer to and follow the QA/QC published in th e “equivalent” EPA method for the parameter, second the laboratory may refer to the appropriate QA/QC section(s) of an approved 40 CFR Part 136 method from a consensus organization compendium, or third, the laboratory may incorporate 12 QC elements into their documented SOP for performing compliance an alyses when using an approved 40 CFR Part 136 method that lacks such procedures. The 12 QC elements are:

 Demonstration of capability;

 Method detection limit (MDL);

 Reagent blank (also referred to as a method blank);

 Laboratory fortified blank (also referred to as a sp iked blank, or laboratory control sample);

 Matrix spike (MS), matrix spike duplicate (MSD), or laboratory fortified matrix (LFM) and LFM duplicate for suspected difficult matrices;

 Internal standards (for GC/MS analyses), surrogate standards (for organic analysis), or tracers (for radiochemistry);

 Calibration (initial and continuing);

 Control charts (or other trend an alyses of quality control results);

 Corrective action (root cause analysis);

 QC acceptance criteria;

 Definitions of preparation and analytical batches that may drive QC frequencies; and

 Minimum frequency for conducting all QC elements. (40 CFR 136.7).



### 6.3.8 Analytical Data Management

Analytical data management includes data review and reduction. These steps may be conducted by ERG (as directed by the EPA WAM) or another contract or (e.g., SCC). When ERG is not the contractor providing the analytical data review, ERG receives da ta review narratives and analytical results from EPA. Analytical data review and reduction st eps are presented in the subsections below.



### 6.3.8.1 Analytical Data Review

Analytical laboratories will deliver electronic da ta packages consisting of MS Excel® spreadsheets containing analytical results and documentation packag es consisting of traffic reports and sample receipt information, laboratory analysis sheets and checklis ts, instrument data, and analyte chromatograms, where applicable, to ERG or anot her contractor for data review.

When directed by EPA to perform the analytical data review, ERG will develop and use method-specific checklists to review, verify, and validate chemical and/or biological analysis results and supporting documentation provided by the laboratory. (Attachme nt G presents an example method-specific checklist developed by ERG) ERG will perform this review to ensure data quality, note deficiencies, and, where appropriate, initiate corrective actions with the la boratories. Prompt initiation of corrective actions maximizes the amount of usable data generated from each sampling episode.Revision No. 1 Date: October 2013 Section No. 6 Page 88 of 109

The checklists provide method-specific QC objectives for each parameter in the review (e.g., acceptable instrument signal-to-noise ratio, use of correct spik ing concentration, acceptable recovery ranges). Each checklist will contain the following standard sections listing the detailed criteria that ERG will use to verify and validate results of chemical and/or biological analyses:

 General Review of Sample Identity and Handling – ERG will verify sample identity and traceability. ERG will confirm that all data required for data review, verifi cation, and validation are contained in the laboratory data package and will review information in the data package to verify that the laboratory identified and handled samples properly;

 Review of General QC Requirements – ERG will evaluate the common QC requirements in all methods that include sample preparation, di gestion, and/or extraction procedures; and

 Detailed Review of Method-Specific Parameters – ERG will perform method-specific QC validation, which includes reviewing the prim ary instrument calibrations and performance evaluation samples.

After an analytical data package has been reviewed and all problems that can be corrected have been resolved, ERG will prepare a written data review na rrative. ERG will submit data review narratives, completed checklists, along with a hard-copy summa ry data package, to EPA. The purpose of the narrative is to document the samples and analyses th at were reviewed, summarize the criteria by which they were reviewed, note any deficiencies in the data, and provide the EPA WAM with recommendations concerning data usability. Data deficiencies will be noted in the data review narratives and reported to the EPA WAM. The EPA WAM will determine whether a ny corrective actions should be performed to address the deficient data, such as reanalysis by the la boratory (if sample volume is stored and available) and/or repeated sampling by ERG or the facility and r eanalysis. As a result of data deficiency correction or acceptance, analytical data are typically excluded only in cases of grossly deficient quality control data (e.g., laboratory documentation is inadequate to ev aluate quality control measures, or quality control samples results differ significantly from method requirements).

ERG will use standard language to communicate problems and recomme ndations. Table 6-4 lists potential data limitations and associated qualifiers that ma y be used in EPA’s sampling programs. Any data qualifiers will be fully described in the final laboratory packages and data review narratives.

Table 6-4. Data Quality Qualifiers for Sampling Program

Potential Data Limitations Associated Qualifiers Example Data Qualifier Flag Holding time exceeded (for analytes subject to loss or degradation). Sample analyzed outside holding time. H

OPR outside of the performance specifications. Data are of acceptable quality but may reflect either minimum or maximum values O

MS/MSD recoveries above or below performance specifications. Data are of acceptable quality but may reflect either minimum or maximum values. RRevision No. 1 Date: October 2013 Section No. 6 Page 89 of 109

Table 6-4. Data Quality Qualifiers for Sampling Program

Potential Data Limitations Associated Qualifiers Example Data Qualifier Flag Surrogate recoveries above or below performance specifications. Data are of acceptable quality but may reflect either minimum or maximum values. S

Sample results greater than five times but less than ten times blank result. Data may reflect maximum value L

Sample result greater than upper end of calibra tion range. Include the value corresponding to the upper limit of the calibration range in the database with a “>” sign. E

Sample result less than five times blank result. Include the value in the database as not detected at the adjusted method detection limit. ND

Duplicate or MS/MSD relativ e percent difference greater than specified limit. Data provided represent the best obtainable data. T

Grossly deficient quality control data. Exclude data from database (with EPA WAM concurrence). Data Qualifier Flag Not Applicable



### 6.3.8.2 Analytical Data Reduction

Each laboratory assigned to analyze wastewater samp les under this contract will ensure accurate and contract-compliant data prior to release of the da ta. The laboratory will cross-reference all of its laboratory QC data to sampling episode-specific anal yses. Documentation of the laboratory QC analyses is included in the laboratory data package. ERG w ill take the analysis results from the laboratories (reported by sample number) and correlate them to th e appropriate trip blank, field blank, equipment blank, method blank, field duplicate, MS, MSD, and/or field duplicate analysis in the SER. All dates pertinent to the sample episode (e.g., dates on whic h each sample is collected, shipped, received by laboratory, prepared, and analyzed) will be referenc ed to the appropriate analytical holding times.

The analytical laboratories will perform the analytical data reduction. The laboratories will monitor data reduction using results produced for internal standa rds, surrogates, laboratory duplicates, and spikes.

The laboratories will each perform quantitation acco rding to protocols specified in laboratory subcontracts in one of the following ways:

 By using the isotope dilution technique;

 By using the internal standard technique;

 By using the external standard technique;

 By using known response factors specified in the methods;Revision No. 1 Date: October 2013 Section No. 6 Page 90 of 109

 By comparing the instrument response to a built-in scale;

 By titration to a reaction-defined endpoint; or

 By comparing instrument response from a treated sample to a calibration curve generated from instrument responses to known concentration standards.

Results will be reported in units of we ight/volume (such as micrograms per liter ( μ g/L) or milligrams per liter (mg/L)) for aqueous samples, in units of weight/w eight (on a dry-weight basis) for solid samples, or in other units as specified by the method or the la boratory subcontract. Laboratory data results are provided either to EPA (who will then forward them to ERG) or to ERG directly. Section 6.3.6 describes ERG’s procedures for putting laboratory results into a database format for presentation and analysis.



### 6.3.9 Analytical Data Reporting

Upon completion of data review, reduction, and validation procedures for all samples and analytes included in a sampling episode, ERG will prepare a database that summarizes analytical results in accordance with the database development procedures d escribed in Section 8.0. To increase efficiency among data users, ERG uses a standard database form at for all industry studies. Table 6-5 shows the typical structure of database files and source of data. The structure of the database files may be reviewed and changed as appropriate for a specific program.

Table 6-5. Typical Analytical Database Structure

Field Description Field Name Field Length Source of Data Analyte Concentration AMOUNT 16 Laboratory Analyte Name ANALYTE 75 Laboratory CAS Registry Number CAS_NO 10 Laboratory Calculated Detection Limit DETLIMIT 8 Laboratory Dilution Factor DILUTION 8 Laboratory Episode Number EPISODE 4 Upper Ri ght Corner of Traffic Report Laboratory Name LABNAME 5 Laboratory Column or Extraction Type PROCEDUR 10 Laboratory Reason for Qualification REASON 60 Laboratory Sample Number SAMPLE 5 Left Column of Traffic Report Data Qualifier Flag QUAL 11 ER G/Other Contractor Data Review Concentration Units UNIT 7 Laboratory Laboratory Method Used METHOD 12 Laboratory

To communicate the quality of individual results, ER G chemists (when ERG is conducting the analytical data review) or other data review contractors will represent the data qualifier information as flags that indicate the result of the data review. Examples of typical data qualifier flags are shown in Table 6-4.

ERG will retain all sample collection records in the files for the project. These records will include sitespecific sampling plans, field sampling log sheets, field data sheets, preservation log sheets, and trafficRevision No. 1 Date: October 2013 Section No. 6 Page 91 of 109

report forms. Copies of these records will be provi ded to EPA as requested. Laboratory result packages (and laboratory data review packages if ERG is conducting the data review) will also be retained with the project files.

At the completion of each sample episode, ERG prep ares a draft SER that includes descriptions of the facility, wastewater sources, wastewater treatment, sa mpling points and sample collection, and deviations from site-specific and generic sampling plans. After recei pt of all analytical results (including the data narratives), ERG will prepare a final SER for each sampling episode. The final report will provide descriptions of the facility, wastewater sources, wastewater treatment, sampling points and sample collection, deviations from site-specific and generi c sampling plans, summary tables of the analytical results, and discussion of the quality control results (inc luding field, equipment, and trip blanks; sample duplicates; and the results of the MS/MSD). The re port will also present in-field measurement data and engineering data provided by the facility.

As noted in Section 6.3.8.1, an y data deficiencies are identified and discussed in the data review narratives, including any data excluded from the analy tical database due to grossly deficient quality control data. The SER will not include excluded data and it is the EPA WAM who decides whether other flagged data are usable for the purposes of the sa mpling program. The data review narratives are maintained as part of the project record and are includ ed as an attachment to the final SER if directed by the EPA WAM.

Information collected through sampling episodes cl aimed as CBI by sampled facilities will be handled according to the procedures outlined in ERG’s CBI security plan entitled Security Plan for Handling Confidential Business Inform ation Under the Clean Water Act Contract EP-C-12-021 , dated November 9, 2012 and approved by Ahmar Siddiqui, EPA’s Docu ment Control Officer, on January 9, 2013. Only staff demonstrating their understanding of CBI ha ndling procedures by passing a written test will be permitted to access CBI.



### 6.4 Achieving the Contract’s Overall Quality Objectives

ERG’s field sampling and QC procedures described in Section 6.1.5 and an alytical QC procedures described in Section 6.3.4 are specifically designed to provide EPA with analytical data of known quality and integrity. EPA’s main goal in collecting primary data through sampling and analysis is to identify information that is specific to an industry and/or technology of interest and thus usable for EPA’s
purposes.Revision No. 1 Date: October 2013 Section No. 7 Page 92 of 109



### 7.0 G ENERATION 

### , E VALUATION 

### , AND 

### / OR 

### A NALYSIS OF 

### D ATA 

### U SING 

### S PREADSHEETS

ERG uses both existing data and primary data coll ected or generated under Contract EP-C-12-021 to conduct technical evaluation and analysis in support of EAD’s industry studies and ELG development. In support of those efforts, ERG will often use spreadsheet s to collate, compare, calc ulate, and analyze data. Because use of spreadsheet tools (usually MS Excel®) is integral to much of our work, ERG has developed standard procedures for developing, documenting, and evaluating the accuracy of spreadsheet data with the purpose of providing EPA with data of known quality a nd integrity. This section discusses ERG’s QA/QC procedures specific to spreadsheet development and use along with the staff responsible for each of these steps.



### 7.1 Developing Spreadsheets

A spreadsheet is an electronic table that is used to enter, arrange, calculate, and analyze data. ERG uses spreadsheets to store and manipulate data as well as to present data in a report or table format. Planning for spreadsheet development usually occurs in c oordination with the EPA WAM and should reflect the overall technical approach that the EPA WAM and ER G WAM have outlined for the task. A spreadsheet developer is the person who creates, names, and uses the spreadsheet; it is the developer’s responsibility to ensure that data in the spreadsheet are imported or entered correctly, clearly identified and labeled, properly referenced, and that mathematical functions used for manipulating data are executed accurately. ERG maintains version control for spreadsheets by incl uding the date or version number in the file name.



### 7.2 Quality Control and Data Quality Goal

All project spreadsheets undergo technical review by technical/project staff ot her than the spreadsheet developer. The responsibilities of the technical reviewer are to verify that the technical approach and procedures used in the spreadsheet are reasonable a nd logical, verify that the spreadsheet documentation is complete and clear, and verify that calculations and results are accurate by manually verifying equation cells.

The ERG WAM will work with the EPA WAM, project staff developing a spreadsheet, and the technical reviewer to determine the level of QC necessary (i.e ., whether all data points and calculations must be checked or whether only a percentage should be checked ). For example, a spreadsheet column may have a repeating calculation that runs the length of the colu mn rows and a check of the calculation for one row will serve to ensure accuracy for all rows. Depending on the spreadsheet size and work assignment/project budget and time constraints, it may be appropriate for ERG to initially check only a percentage of large spreadsheets. If, through the per centage check, spreadsheet corrections are identified, ERG will address corrections for those columns or rows identified and any others that may be impacted by the same issue. ERG will work with the EPA WAM to determine whether expanded QC checks are necessary. The technical reviewer will identify any erro rs or deficiencies in data labeling and references to the spreadsheet developer who w ill implement the needed corrections.

ERG’s data quality goal is for 100 percent accuracy of the data in the spreadsheets developed on this contract. ERG’s spreadsheet review is designed to id entify and correct any data errors to achieve this goal.Revision No. 1 Date: October 2013 Section No. 7 Page 93 of 109



### 7.3 Documenting Spreadsheet Quality Control

ERG staff will document technical review of spread sheets using an electronic Deliverable Sign-Off Sheet (see Attachment H) or QA worksheet. As deemed necessary, the technical reviewer will provide a written summary of the data checked, errors or problem s found, and recommendations for revisions. This summary may be provided separately to the spreadsheet developer or included in a QA worksheet in the spreadsheet workbook.



### 7.4 Identification and Control of CBI in Spreadsheets

The spreadsheet developer is respons ible for determining if any information used in the spreadsheet is CBI. If so, s/he follows ERG’s CWA CBI Security Plan procedures during spreadsheet development, review, and distribution. All electronic spreadsheet files containing CBI are stored on ERG’s CBI server in dedicated and access-controlled project areas. CBI spreadsheet files are identified as containing CBI by including “CBI” in the file name and including an electronic CBI cover sheet at the front of the spreadsheet workbook.



### 7.5 Quality Assurance

It is the responsibility of the ERG WAM and Project QA Coordinator to require and check that ERG’s spreadsheet development procedures described in th is section are followed. The ERG WAM will specify the necessary quality control for spread sheet development as part of upfro nt task planning. In addition, deliverable review records or QA worksheets provide documentation for the EPA WAM that spreadsheet QC occurred. The Project QA Coordinator will support pr oject staff in the use of the spreadsheet SOP and will be able to verify that QC is occurring when they audit project quality procedures by reviewing QC documentation.Revision No. 1 Date: October 2013 Section No. 8 Page 94 of 109



### 8.0 G ENERATION 

### , E VALUATION 

### , AND 

### / OR 

### A NALYSIS OF 

### D ATA 

### U SING 

### D ATABASES

ERG uses both existing data and primary data coll ected or generated under Contract EP-C-12-021 to conduct technical evaluation and analysis in support of EPA’s industry studies and ELG development. In support of those efforts, ERG will often use databases to store, organize, calculate, and analyze data. ERG uses databases to store and analyze questionnaire responses, sampling results, long-term treatment performance data, and for development of cost, load ing, and environmental impact calculation models. Because use of database tools (usually MS Access®) is integral to much of our work, ERG has standard procedures for developing, documenting, and evalua ting databases and database queries with the purpose of providing EPA with data of known quality and integrity.. This section discusses quality control and quality assurance procedures specific to databases and database queries along with the staff responsible for each of these steps.



### 8.1 Developing Databases

A database is a collection of data that is organized so that its contents can easily be accessed, managed, and updated. The most prevalent type of database is th e relational database, which is a tabular database in which data are defined so that they can be easily reorganized and accessed for data calculations, with a shared attribute providing a relationship between da ta tables. Planning for database development will occur during the planning stages of a task. Examples include database development for a questionnaire/survey effort and when planning an a pproach for developing industry costs and loads to evaluate baseline and regulatory option loads, load removals, and industry costs. The ERG WAM will work in close coordination with the EPA WAM when planning these activities as part of the systematic planning process for the task and is responsible fo r reviewing and approving th e written plan for overall development and quality control of the database(s).

The ERG WAM will identify a database task leader responsible for the data base development and analysis effort. In some cases, the task leader will work with a datab ase developer or the task leader will also be the developer. The database developer is responsible for selecting the appropriate database software, describing the database development plan, a nd working with the task leader to assign staff to review, data enter, and QC project data that are included in the database. Data entry for databases may include manual input of data into forms or tables or importing of datasets into the database. The task lead will also be responsible for ensuring that all supporti ng documentation for the information in the database is referenced and that documentati on of the final database is prepared. ERG maintains version control for databases by including the date or version number in the file name.



### 8.2 Developing Database Queries

Database queries are used to view, change, and analyze data contained in one or more data tables. They are constructed either directly using structured que ry language (SQL) or indirectly with a user-friendly interface. There are two types of queries: select and action. Select queries retrieve a result from a set of records in one or more tables. Action queries ask fo r additional operations on the data, including creating new data, adding to existing tables, updating data, and/or deleting data . In addition to database queries, database macros may also be used as a shortcut to task activities that are perfo rmed repeatedly. Examples include opening a report, running a query or set of queries, or closing the database. Another common feature of databases are database modules, which ar e a collection of declarations, statements, andRevision No. 1 Date: October 2013 Section No. 8 Page 95 of 109

procedures that are stored together as a unit, similar to a macro. ERG constructs database modules using Visual Basic for Applications (VBA) programmi ng language, which can allow for a more complex combination of procedures to retrieve, change, or manipulate data. A module can be either a class module or a standard module. Class modules are attached to forms or reports and contain procedures that are specific to the form or report. Standard modules cont ain general procedures that are not associated with any other object.

A query developer is a person who creates, names, and uses database queries, macros, or modules to develop results specified by the database task lead an d the database plan. The query developer designs the desired actions and mathematical functions, creates a query documentation table in the database, ensures that their queries have been reviewed, and addresses any comments from technical review.



### 8.3 Quality Control and Data Quality Goal

All project databases and database queries undergo te chnical review by staff familiar with database programming. The responsibilities of the technical reviewer are to verify that the technical approach and database design structure and queries are reasonable and logical; verify that original data are input correctly; verify that queries, calculations, and resu lts are accurate; verify that sequential queries are working properly; and check that database documen tation is complete and clear. The ERG WAM will work with database task leader and technical reviewer to determine the level of QC necessary. If needed, the Project QA Coordinator should be consulted for guidance in determining effective ways to conduct the QC. The Project QA Coordinator will audit database development to ensure that all required reviews have been conducte d and documented.

ERG’s data quality goal is for 100 percent accuracy of the data put into or generated by the databases developed on this contract. ERG’s database and da tabase query reviews are designed to identify and correct any data errors to achieve this goal.



### 8.3.1 QC for Database Data Entry

Procedures for ensuring quality control in data entry include the following:

 Plan a consistent method for data entry.

 When manually entering data, use double key entry where possible. This method allows for comparison of the two entries to identify errors and correct them.

 When importing data, check that data fields have been successfully imported, ensure that zero and blank values are handled appropriately, and ensure that data truncation has not occurred due to setting field sizes too small or because th e correct field type was not selected.

 Count records to ensure the number of records in each table is correct.

 For data entry input forms, have another team member or programmer review the programming code to ensure that data are saved in the correct tables and formats.

 Document any data entry reviews in the database documentation table.Revision No. 1 Date: October 2013 Section No. 8 Page 96 of 109



### 8.3.2 QC for Database Queries and Outputs

Procedures for ensuring quality control in database queries and database outputs include the following:

 Verify record counts;

 Check equations for accuracy by performing hand calculations to compare to outputs;

 Confirm that constants are correct;

 Check that items are jo ined appropriately; and

 Manually check records.



### 8.4 Documenting Database Quality Control

ERG staff will document technical review of da tabases and their underlying database queries. QC documentation may be maintained in the database itself (as part of the database documentation table) or in separate documentation tables (usually MS Excel® files). Database QC documentation will include resolution of double-key entry comparisons (for manu al data entry), database review revisions and corrections, and documentation of QC for queries.



### 8.5 Identification and Control of CBI in Databases

The database task leader and database developer are re sponsible for determining if any information used in a database is CBI and, if so, following ERG’s CWA CBI Security Plan procedures during database development, review, and distribution. The ERG data base developer will identify CBI data included in a database with a CBI designation tag. A typical exampl e of this is a database composed of questionnaire data where some data are identified as CBI by th e questionnaire respondents and some data are not. ERG’s questionnaires are designed with CBI designati on boxes for each question, part of a question, and/or comment, which then provides a CBI designation tag once the data are entered or imported into a database. In this case, the database table containi ng questionnaire responses includes a column for the data and an adjoining column with CBI designati on (yes/no). Any database containing CBI data will be developed, reviewed, maintained, and stored on ERG’s CBI server in dedicated and access-controlled project areas. Databases containing CBI data are not released to the public until and unless they are sanitized (i.e., all CBI data and CBI-de ducible data have been removed).



### 8.6 Quality Assurance

It is the responsibility of the ERG WAM and Project QA Coordinator to require and check the use of ERG’s database and database query quality control pr ocedures. The ERG WAM, in consultation with the database task leader and Project QA Coordinator, w ill specify the necessary level of quality control for database development as part of upfront task pl anning. The ERG WAM will review database quality control documentation to ensure that QC occurred. The Project QA Coordinator will support project staff in use of the database quality control procedures and will verify that QC is occurring when s/he audits project quality procedures by reviewing QC documentation.Revision No. 1 Date: October 2013 Section No. 9 Page 97 of 109



### 9.0 G ENERATION 

### , E VALUATION 

### , AND 

### / OR 

### A NALYSIS OF 

### D ATA 

### U SING 

### M ODELS

In support of industry studies and ELG developm ent projects under Contract EP-C-12-021, ERG may create MS Excel® or MS Access® calculation models for:

 Developing point source category costs, pollutant reductions, and/or non-water quality impacts for regulatory options under evaluation by EPA;

 Modeling environmental impacts (such as receiv ing water concentrations and wildlife tissue values); and

 Modeling other impacts of regulatory opti ons under evaluation as requested by EPA.

Additionally, ERG may identify and use existing mode ls created by EPA, other agencies, or other entities to support our technical analyses conducted under this contract. This section identifies ERG’s procedures for generating and evaluating data us ing models and the quality criteria ERG will use to ensure that the use of these models is appropriate.



### 9.1 ERG Generated Computational Models

Under this contract, ERG will create MS Access® databases or MS Excel ® spreadsheets, write computational or module code, and create queries to develop:

 National-scale compliance modeling tools to est imate the cost to comply with proposed regulatory options and estimate expected pollutant reductions and non-water quality impacts – in support of compliance cost and polluta nt reduction modeling analyses; and

 National-scale screening-level environmental modeling tools to estimate receiving water concentrations and evaluate wildlife tissue valu es required to meet project objectives outlined in Section 2.2 – in support of environmental modeling analyses. ERG will follow our quality control procedures fo r spreadsheets and databases in developing these models; Sections 7.0 and 8.0 provide details on ERG’ s QC and assessment procedures for development of spreadsheets, databases, and database queries. ERG documents the following elements of our computational models in project -specific methodology memoranda:

 Model description;

 Flow charts (if necessary);

 Database description;

 Data sources;

 Documentation of assumptions;

 Discussion of any boundary or limiting conditions;

 Documentation of default parameter values and their sources; and

 Documentation of all equations (and sources, if a pplicable) used by the model to generate output.

ERG will document code developmen t and modifications for each version of the model. Specific documentation that will be kept for each version will include the following:Revision No. 1 Date: October 2013 Section No. 9 Page 98 of 109

 ERG’s code modules or database queries create d to manipulate data will include inline programmer comments or documentation in a separat e file describing the purpose and equations used in the query;

 ERG will maintain a table (in the database, as a tab in the spreadsheet, or as a separate file) that documents/tracks significant changes made to the model; and

 ERG will track errors identified and subsequent resolution in the database, as a tab in the spreadsheet, or as a separate file.

ERG will assign a technical reviewer to review and verify module code and query code to check the correctness and accuracy of the governing equations and to assure that the database code is fully operational. ERG will verify that input files and cha nges to model input parameters are correct using our established technical review process as outlined in Sections 7.2 and 8.3.

ERG will test each version of the model prior to delivery to EPA. Test cases will be created for each version of the model and will consist of a predet ermined sequence of steps to be executed and the expected output or results. ERG will establish speci fic test case criteria and testing sequences in consultation with the EPA WAM. ERG will execute a nd document the test cases once test case criteria are established.

ERG defines model calibration as the process of adjusti ng model parameters within physically defensible ranges until the resulting predictions give the best possi ble or desired degree of fit to the observed data. ERG defines model verification as the comparison of predictions between a calibrated model and available data that were not used in the model de velopment and calibration. ERG will use available cost, loading, and stream monitoring data at select sites from secondary sources to calibrate and verify that model estimates are reasonable. ERG will consult with expert scientific researchers who are knowledgeable about model subject matter to review model output for reasonableness. The desired degree of fit will be determined on a case-by-case basis and in conjunction with the model verification. Model accuracy is usually defined in terms of percentage difference between modeled outputs to known values. ERG will consult with the EPA WAM on any modeling efforts to determine whether the model output estimates are reasonable for the intended purpose.



### 9.2 Existing Models

In support of industry studies and ELG developmen t under this contract, EPA and ERG may identify existing models for use that support evaluation of costs, loads, or environmental impacts. ERG’s procedures for assessing the quality of the data used in these models, the validity of the model calculation methodology, and the usefulness of modeled outputs are presented below. Before use of any existing model, ERG will assess the following:

 Does the model support answering the questions posed in Section 2.2?

 Is the origin of the model and a description of its inputs, modeling methodology, and outputs documented and available?

 Has the model been evaluated and accepted for use through an EPA Peer Review process or other expert review process?Revision No. 1 Date: October 2013 Section No. 9 Page 99 of 109

 Is the intended use of the model appropriate to the point source category under review by EAD?

ERG will consult with the EPA WAM on the answ ers to these questions and the EPA WAM will determine whether the use of the model is appropriate.



### 9.2.1 Review of Data Quality in Existing Models

An existing model will require model input as well as pre-existing model data. ERG will apply the same data quality acceptance criteria to existing models as those applied to existing data. Section 4.3 discusses ERG’s data quality criteria and acceptance criteria for existing data.

Data quality acceptance criteria for input data used in existing models will include the following:

 Preference will be given to primary data sources (e.g., EPA data collected directly from the point source category under investigation);

 Data sources that are publicly available and ge nerated by other federal Agencies (e.g., USGS databases, National Oceanic and Atmospheric Admi nistration databases) will be considered of sufficient quality for use; and

 Data sources authored by members of industry (e.g., trade association documents or reports authored by industry committees) or other existi ng data sources will be considered for use and, where possible, checked for comparab ility against other data sources.



### 9.2.2 Review of Model Methodology

ERG will utilize the following review criteria wh en assessing existing models for use in ELG development:

Reasonableness of the Model Methodology . ERG will review model docum entation to understand and assess the calculations and assumptions used. Models without documentation on calculation methodology and underlying assumptions will not be acceptable for use. ERG will discuss existing model assumptions with the EPA WAM to determine whether they are acceptable for the intended use.

Model Sensitivity . ERG will test existing models using a range of input data to assess the sensitivity of the model to the data. ERG will discuss the results of sensitivity testing with the EPA WAM.

Representativeness . ERG will review the intended purpose and use of existing models to assess whether they are appropriate for EPA’s assessment of data in support of an ELG. Models whose purpose and circumstances do not represent the industry, pollutant s, technologies, and/or environment under study by EAD should not be used.

Accuracy and Comparability of Model Outputs . ERG will evaluate the accuracy and comparability of modeled outputs by comparing them to existi ng industry data measuring or representing the same intended values. ERG will calculate a relative pe rcent difference between modeled values and known measured values and review these differences with the EPA WAM.Revision No. 1 Date: October 2013 Section No. 9 Page 100 of 109



### 9.3 Additional Resources

ERG will develop, as directed, a project-specific QAPP to address ERG development and use of any environmental models not covered by Section 9.1. ERG will use EPA’s Guidance on the Development, Evaluation and Application of Environmental Models (CREM, 2009) when developing and applying such models. In addition, EPA Region 1 has developed a QAPP template specific to environmental modeling which can be accessed at http://www.epa.gov/region1/lab/qa/qamodeling.html . ERG will use this template as a basis for developing the project-sp ecific QAPP which will detail the purpose of the model developed, the source and quality requirements for m odel inputs, the basis of the modeling methodology, data management procedures, and procedures for mode l calibration, model validation and simulation, and reconciliation with the user requirements.Revision No. 1 Date: October 2013 Section No. 10 Page 101 of 109



### 10.0 D ELIVERABLES 

### R EVIEW

ERG will generate written deliverables as part of work conducted under Contr act EP-C-12-021. Types of deliverables will include:

 Work plans;

 Quality-related documents (QMP, PQAPP, quality documentation);

 Planning and status memoranda;

 Technical letters or memoranda;

 Contact documentation (e.g., telecons and emails);

 Study reports;

 Survey instruments;

 Site visit reports;

 Sampling and analysis plans;

 Health and safety plans;

 Sampling episode reports;

 Technical development documents (including cost and loading reports);

 Environmental and/or econom ic assessment documents;

 Response-to-comment documents; and

 Technical progress reports.

This section presents an overview of QA/QC procedur es that ERG uses to ensure the quality of written deliverables.



### 10.1 Levels of Review

ERG has an established internal QC review pro cedure for work assignment deliverables (memoranda, technical and engineering reports, background documents , etc.). Table 10-1 shows the levels of review that ERG requires for various types of deliverables. At the direction of the EPA WAM, ERG will deviate from these levels of review to accommodate situa tions where there is limited time and/or budget.

ERG conducts technical tasks and prepares deliverables us ing: 1) a conceptual review, 2) a developmental review, and 3) a final product review.

A conceptual review is performed during the initial stages of work development and ensures that the final product and associated documentation address the needs set forth by the EPA WAM and work assignment. Conceptual review will be provided by senior managers knowledgeable in the work performed but not directly involved in the work assignment.Revision No. 1Date: October 2013Section No. 10Page 102 of 109Table 10-1. Levels of ReviewWork ProductTeamMember/TechnicalReviewer1ConceptualReviewer2Project QACoordinatorERGWAM3TechnicalEditor4ProgramManagerInternal project documentation XPlanning documents/Methodology development X X XSpreadsheets and Calculations X X7Assumptions and Conclusions X X XDesign of database, Data operations, and Models X X7XDatabase queries and Modeling runs XClient deliveries that include data or technical analyses X XWork Plan and Cost Estimate X6XQA Project Plan (QAPP)5X X XDocumentation to Client on Project Status, Financial Status,Additional Funds Required, Invoices, etc.X XReports X X XInternal project quality audit X8X1Review by a team member familiar with the work that did not perform the task, often the task leader.2The conceptual review is conducted by the Project QA Coordinator or other qualified senior staff member who is not directly involved in the work assignment.3Informal emails and transmittal letters to client with no datapresented will be reviewed by the ERG task lead. As needed, theERG WAM may designate analternate reviewer to accommodate delivery schedules.4Unless otherwise directed by EPA, first drafts of documents willnot be reviewed by a technical editor. ERG will conduct a technical edit of documents afterincorporating EPA’s comments.5Meets EPA QA/R-5 requirements.6ERG WAM review of a work plan and cost estimate occurs when these deliverables are prepared by a team member/task leader otherthan the ERG WAM.7The Project QA Coordinator checks that spreadsheet and database development QC procedures are followed by reviewing QC documentation during audits.8The Project QA Coordinator will review internal project audit reports when they are prepared by a team member other than the Project QA Coordinator. In mostcases the Project QA Coordinator will conduct internal project audits for the project.Revision No. 1 Date: October 2013 Section No. 10 Page 103 of 109

The quality of intermediate deliverables and final products is also evaluated as these work products evolve. This developmental review includes: 1) a check on calculations and data (see Sections 7.0 and 8.0 for QC conducted on spreadsheets and databases), and 2) a review of draft documents to ensure that the direction of work is consistent with the conceptual review outline. The ongoing status of work tasks will also be reviewed weekly by the project team and reported to the EPA WAM.

Final product review is conducted on all deliverables prior to delivery to EPA.



### 10.2 Types of Review

For developmental and final reviews on written deliver ables, ERG conducts the follow types of review:

 Technical review;

 Copyedit review; and

 Technical edit review.

ERG determines the type of document review needed based on the type of deliverable , its status (draft or developmental stage versus final), and the directi on of the EPA WAM. Draft or developmental reviews will include a technical and copyedit review. All fi nal major documents (such as Technical Development Documents and Final Study Reports) will undergo both technical review and technical editing. The extent of final deliverable editing will be dictated by the EPA WAM because it may depend on time and resources available as well as the desire by EPA to main tain specific text that has been developed through EPA internal review and shoul d not be subject to change.



### 10.2.1 Technical Review

A technical review is conducted by senior staff w ith sufficient technical background and experience to understand and critique the work described in the t echnical memoranda, report, or document but who did not author the deliverable. The technical reviewer will assess the following:

 The organization of the deliverable;

 Whether the technical presentation of data is clear, logical, and sources are documented;

 Whether any results, conclusions, and recommendati ons based on the technical analysis presented are substantiated by the data and arguments presented; and

 Whether the writing is clear and understandable.

A technical reviewer may also identify copyediting or substantive technical editing issues (as listed below).



### 10.2.2 Copyediting Review

Copyediting is generally conducted by members of ERG’ s editorial team but may also be conducted by other project staff. A copy editor evaluates the deliv erable for micro-level issues (but will note macrolevel issues for a document manager to consid er). Examples of micro-level issues include:Revision No. 1 Date: October 2013 Section No. 10 Page 104 of 109

 Style (e.g., follows ERG or EPA style guide for the type of document);

 Formatting (e.g., levels of headers, TOC/page numbers/graphics issues);

 Punctuation;

 Grammar and syntax;

 Typos; and

 Consistency (e.g., in use of acronyms).



### 10.2.3 Technical Edit Review

ERG’s editorial staff will also provide, as needed, substantive technical editing of final documents. Technical editing includes copyediting as well as ev aluation of macro-level issues such as:

 Identification of illogical thinking or poor organization;

 Identification of difficult-to-understand material;

 Appropriateness for the target audience or goal of the document;

 Correction of poorly constructed sentences;

 Review of word choice and sentence corrections to smooth awkward spots;

 Review of passive versus active voice; and

 Identification of redundancy.



### 10.3 Quality Assurance for Deliverable Review

As described in Table 2-4, project staff authoring project deliverables are responsible for following the quality control procedures described in this PQA PP and any project-specific QAPP or SQAPP. The author of a deliverable will coordinate review and will work with the ERG WAM as needed to determine the appropriate levels of review and identify revi ewers. The project deliverable author will then incorporate any needed edits or re visions based on internal review. After receipt of any EPA review comments on a draft deliverable, the ERG project de liverable author will again be responsible for addressing and incorporating any edits or revisions.

The ERG WAM is responsible for ensuring that project staff comply with the necessary quality control reviews and will coordinate with the EPA WAM, when needed, to respond to any requested deviations from the Table 10-1 levels of review. ERG’s Project QA Coordinators assess the project for compliance with our stated quality control procedures through part icipation in project team meetings, participation in determining corrective action if quality issues arise, and through project audits as described in Section 11.1.Revision No. 1 Date: October 2013 Section No. 11 Page 105 of 109



### 11.0 ERG’ S 

### Q UALITY 

### P ROGRAM 

### A SSESSMENTS AND 

### R ESPONSE 

### A CTIONS

ERG’s quality program for Contract EP-C-12-021 includes the following main components:

 Development of a customized QMP for the contract;

 Development of this PQAPP;

 Designation of a Contract QA Manager (Mary Wille tt) and Project QA Coordinators to support development of project-level QA/QC procedures to use with each work assignment; and

 Activity-specific SOPs.

This section discusses how ERG ensures that our QA program is implemented and ERG’s response actions to address quality concerns or deficiencies.



### 11.1 ERG Quality Program Assessments

The responsibility for assessing the use of ERG’s quality pr ocedures rests with all project staff as follows:

 The ERG Contract QA Manager is responsible for ensuring that quality planning for the contract has been performed; she documents this planning in the customized QMP and this PQAPP. In addition, the ERG Contract QA Manager will ascer tain that our quality control procedures are being implemented by participating in biweek ly ERG WAM meetings where quality issues are discussed and through a more formal auditing of the contract projects on an annual basis using our QA audit checklist (see Attachment I). The ER G Contract QA Manager will review project audit results with the ERG WAM and Program Ma nager and discuss any observations relating to ERG procedures for future work under the contract.

 The ERG Program Manager is responsible for supporting the ERG Contract QA Manager with quality planning for the contract and is also the lead in assigning responsibilities for quality control development and assessment at the wo rk assignment level. The ERG Program Manager will address any substantive quality issues that ar e raised and has the authority to require additional quality control activities, staff training, and adjust project staffing as needed.

 Project QA Coordinators are responsible for s upporting individual work assignment and project teams in implementing quality control and qua lity assessment activities at the work assignment level. The Project QA Coordinator will work w ith the ERG WAM to support development of SQAPPs or individual project QAPPs as need ed and will conduct project-level QA audits as directed by the ERG Contract QA Manager to assess implementation of QC. The Project QA Coordinator will complete an audit checklist a nd discuss audit observations with the ERG WAM and Contract QA Manager at the completion of any audits.

 ERG WAMs are responsible for ensuring that project staff understand the QA/QC procedures they are to implement in conducting the wo rk assignment/project. The ERG WAM will work with the EPA WAM and ERG’s Program Manager to decide on the amount of data checking to perform for spreadsheet and database development and the levels of review required for project deliverables.Revision No. 1 Date: October 2013 Section No. 11 Page 106 of 109

 ERG project staff is responsible for implementing and documenting project quality procedures as specified by this PQAPP, SQAPPs, or project-specific QAPPs, and for using ERG’s SOPs.



### 11.2 Response Actions

Quality assurance response actions are implemented at the work assignment/project level and, when necessary, at the contract level as follows:

Quality Assurance Finding Response Action Roles/Responsibilities
Insufficient Quality Planning The ERG Program Manger and ERG WAM will work with the EPA WAM to identify and incorporate the needed work assignment/project quality planning. ERG will document QC plans in a systematic planning deliverable. The ERG Contract QA Manager and the ERG Program Manager will monitor the contract work assignments/projects to ensure that quality planning is done. The ERG WAM and Project QA Coordinators will conduct quality planning and document plans in a systematic planning deliverable.

QC Procedures Not Followed (includes failure to implement a QC check or failure to document a QC check) The ERG reviewer or ERG WAM will require that project staff to implement the missing QC procedure(s). The ERG reviewer and/or ERG WAM identifies missing QC. The ERG project staff implement the necessary response action.

EPA WAM identifies Quality Concerns The ERG Program Manager and ERG WAM will meet with the EPA WAM to discuss the quality concern and investigate the sour ce of the problem. The ERG WAM will work with the EPA WAM to identify mitigation procedures and will implement fixes to the deliverable. The ERG WAM in consultation with EPA WAM identifies necessary QC and directs project staff. The ERG project staff implement the necessary response action. As needed, ERG Program Manager specifies additional or replacement staff and/or procedures to address the EPA concern.

QA Procedures Not Followed (includes failure to implement QA review or document a QA review) The ERG Contract QA Manager will direct the appropriate Project QA Coordinator to implement the missing QA review and/or documentation. The ERG Contract QA Manager and Project QA Coordinators implement the missing QA and/or documentation.Revision No. 1 Date: October 2013 Section No. 12 Page 107 of 109



### 12.0 A SSESSING 

### D ATA 

### U SABILITY AND 

### R ECONCILIATION WITH 

### U SER 

### R EQUIREMENTS



### 12.1 Assessment of Data Usability

Components of ERG’s assessment of data usability include:

 Data review against quality specifi cations and goals as defined in Sections 3.0 through 8.0 above or in a project-specific QAPP or SQAPP;

 Discussion of ERG’s data quality review results in the context of the work assignment/project with the EPA WAM (via memoranda and/or meetings) ;

 Decisions by EPA on the usability of the da ta and direction from the EPA WAM to ERG regarding any adjustments to the analysis (e.g., a dditional data identification and/or collection, alternative analysis, etc.) based on those decisions; and

 Documentation of data review and usability as part of the project or regulatory rulemaking effort.

Ultimately, it is EPA’s responsibility to make final determinations regard ing data usability. ERG’s role is to provide EAD with recommendations regarding data usability and help reconcile data quality issues as necessary to help facilitate use of data by EPA to meet its needs.



### 12.2 Reconciliation with User Requirements

ERG will clearly identify sources and limitations of exis ting data collected in all project deliverables. EPA WAMs will determine the usability of existing da ta collected under work assignments and provide written documentation of their determinations to ERG and the EAD QA Coordinator. ERG will thoroughly document all calculations and analyses for a work assignment’ s project file in spreadsheets, databases, calculations, database programs and queri es, technical memoranda, and documents. ERG will use only reviewed and accepted data in calculations and analyses, and any data limitations or assumptions will be clearly described to the user. Data produced or developed under contract work assignments will be reconciled with project goals during the technical re view process as described in Sections 3.0 through 10.0.

For the 304(m) annual review process, ERG will comp are findings to known values to verify results, including overall industry trends, pollutant loadings , and geographic scope. For example, ERG will use results from previous years’ findings, includi ng published screening-level databases, the TRIReleases and DMRLoads databases and the most recent Preliminary and/or Final ELG Plans described in Section 3.0 of this PQAPP. Additionally, ERG will report how well da ta collected and analyzed for the 304(m) project answer the key questions identified in Section 2.2 of this PQAPP.

Completed questionnaires and sampling data are primar y data sources that ERG will use to assist EPA in answering, at least in part, each of the key questions posed in Section 2.2 as part of industry studies and ELG development. Questionnaire responses will provide detailed descriptions of facility characteristics, operations, wastewater generation, pollutant presence, and treatment technologies in use. Questionnaire data, in combination with other data sources such as sampling and existing data, will be used in a variety of calculations and analyses throughout an indus try study or ELG rulemaking. ERG will thoroughlyRevision No. 1 Date: October 2013 Section No. 12 Page 108 of 109

document all primary data calculations and analyses for the project file in spreadsheets, databases, calculations, database programs and queries, and tec hnical memoranda and documents. ERG will use only acceptable questionnaire data in calculations and analy ses, and any data limitations or assumptions will be clearly described to the user.Revision No. 1 Date: October 2013 References Page 109 of 109



### 13.0 R EFERENCES

Bicknell, Betsy and TJ Finseth. 2006. Eastern Resear ch Group, Inc. Memorandum to Carey Johnston and Jan Matuszko, U.S. EPA. “Comments Received Rega rding POTW Removals.” September 8, 2006.

Codding, Ellie and Deb Bartram. 2005. Eastern Research Group, Inc. Memorandum to the 2006 Effluent Guidelines Program Plan Docket, “Publicly-Owned Treatment Works (POTW) Percent Removals Used for the TRIReleases2002 Database.” August 12, 2005.

CREM, 2009. Council for Regulatory Environmental Monitoring (CREM). Guidance on the Development, Evaluation and application of En vironmental Models. EPA/100/K-09/003. March 2009.

ERG, 2009. ERG. 304m Project File Management Plan. June 2009.

ERG, 2012. ERG . Environmental Engineering Support fo r Clean Water Regulations Quality Management Plan. June 2012.

ERG, 2012b. ERG. Security Plan for Handling Confidential Bu siness Information Under the Clean Water Act Contract EP-C-12-021. November 9, 2012.

EPA, 2001a. U.S. Environmental Protection Agency. EPA Requirements for Quality Management Plans QA/R-2. Office of Environmental Information. March 22, 2001, reissued May 31, 2006.

EPA, 2001b. U.S. Environmental Protection Agency. EPA Requirements for Quality Assurance Project Plans QA/R-5. EPA/240/B-01/003. Office of Environmental Information. March 2001, reissued May 21,
2006.

EPA, 2002. U.S. Environmental Protection Agency . Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Dissem inated by the Environmental Protection Agency. EPA/260R-02-008. October 2002.

EPA, 2009. U.S. Environmental Protection Agency. Memorandum from Richard Reding, EAD/OST to Regional Quality Assurance Managers. Quality Assura nce and Quality Control Requirements in Methods Not Published by EPA. May 7, 2009.

EPA, 2012. U.S. Environmental Protection Agency. Technical Users Background Document for the Discharge Monitoring Report (DMR) Pollutant Loading Tool. 2012.Attachment A

PUBLICLY-OWNED TREATMENT WORKS (POTW) PERCENT REMOVALS USED FOR THE TRIRELEASES2002 DATABASEAttachment B

COMMENTS RECEIVED REGARDING POTW REMOVALSAttachment C

TELEPHONE CONTACT REPORTAttachment D

304(m) PROJECT FILE MANAGEMENT PLANAttachment E

QUESTIONNAIRE MAILOUT CHECKLISTAttachment F

DATA REVIEW GUIDELINESAttachment G

METHOD SPECIFIC CHECKLISTAttachment H

DELIVERABLE REVIEW SIGN-OFF SHEETAttachment I

QA AUDIT CHECKLIST