Agency

Review Draft 

# Handbook for Developing **Quality Assurance Project**  **Plans**

This document is a **review draft** . This information is distributed solely for the purpose of pre
dissemination review under applicable information quality guidelines. It has not been formall y disseminated by

EPA. It does not represent and should not be construed to represent any Agency determination or policy . It is being circulated for review of its technical accuracy .Agency

Review Draft **FOREWORD** The U.S. Environmental Protection Agency (EPA) developed the Quality Assurance Project Plan (QAPP) as a tool for project managers and planners to document the ty pe and quality

of data needed to make environmental decisions and to describe the methods for collecting and assessing those data
. The development, review, approval, and implementation of the QAPP are part of EPA’s mandatory Quality Program . The EPA Quality Program requires all organizations to develop and operate management structures and processes to ensure that data used in Agency decisions are of the type and quality needed for their intended use
. The QAPP is an integral part of the fundamental principles and practices that form the foundation of the EPA Quality Program . This Han dbook is designed to assist in the creation of QAPPs that address the specifications listed in Annex B of Quality Standard For Environmental Data Collection, Production, a nd Use By E PA Organizations (

EPA CIO 2106
- S

- 01 ) and Quality Standard For Environmental Data Collection, Production, a nd Use By Non
-EPA (External) Organizations (EPA CIO 2106
- S
- 02.0 ) ( current version s )
. It is intended both for EPA organizations and for organizations conducting environmental data operations under external agreements with EPA
. This document:  p rovides technical and policy advice;  provides implementation information;  d oes not substitute for statutes EPA administers or their implementing regulations;  i s not a regulation;  d oes not impose legally
- binding requirements on EPA, states, or the regulated community ;  i s not mandatory
. EPA may :  apply this handbook to any

particular program or project to the extent appropriate in light of site
- specific facts ;  not apply

this handbook to a particular situati on based upon the specific circumstances ; and  update, improve, revise, replace or withdraw this handbook at any time. This document is one of several guidance and handbook documents from the U.S. Environmental Protection Agency, Office of Environmental Information (OEI) Quality Staff. EPA Quality Program documents may be downloaded from the Quality Home Page: http://www.epa.gov/quality /qa_docs.html In many instances, regional or program quality ass urance personnel and project management personnel will be engaged for specific implementation information. These personnel are listed here: http://www.epa.gov/quality /contacts.html Questions regardin g this document or other documents can be directed to the OEI Quality Staff:Agency

Review Draft U.S. EPA OEI/Quality

Staff (2811R) Washington, DC 20460 Phone: (202) 564
- 6830 FAX: (202) 565
- 2441 e - mail: quality @epa.govwww.epa.gov/qualityAgency

Review Draft **PROLOGUE** The Environmental Protection Agency ’s ( EPA ) mission is to protect human health and to safeguard the natural environment

— ai r, water, and land

— upon which life depends
. EPA provides leadership in the nation's environmental science, research, education , and assessment efforts
. In order to accomplish this mission, EPA must ensure that our efforts meet quality standards that insp ire confidence in our conclusions
. The EPA Quality Program provides a framework for ensuring that our products and services meet quality standards that are appro priate for their intended use. Quality Assurance Project Plans (QAPPs) are one component of the EPA Quality Program . Each project subject to the Internal or External Standards (CI O 2106 - S - 01.0 and CI O 2106 - S - 02.0, respectivel y) is required to develop and maintain a QAPP
. QAPPs are intended to help a project team document how they are going to conduct their project
. Each project team has innate ideas about the ty

pe and quantity

of data they

will need, how they ’ll know if the data are “good enough” or not, and how they ’ll “double
- check” to be sure that their results make sense and are meaningful
. This document offers some guidance on how to document those ideas
. It is meant to cover the broad spectrum of activities conducted by the Agency such as setting national standards, pollution prevention, permitting, water and air quality, or human health and ecological risk assessment
. Depending on the organization, the title of the person having quality assurance responsibilities can differ from those used in this guidance. For example, a Quality Assurance Manager in one organization may be a Quality Assurance Director in another. From the contextual use in the guidance, the appropriate person should be recognized. There are some areas of environmental data that are minimally addressed in this version of the Handbook including use of questionnaire survey s and method development studies; later versions of the Handbook will address these. There are three principal chapters: Chapter 2 is for projects requiring the collection of new data to answer the study

question; Chapter 3 is for projects making use of exi sting data; Chapter 4 is for projects involving modeling. Each chapter is self contained and should be used in conjun ction with the Internal and External Standards (specifically , Clauses 7.5

– 7.1 0 , and Annex B)
. Chapters and appendices of greatest utility when creating a QAPP vary with the intent and purpose of the data collection :
 New data for a new project: Chapters 1 and 2, Appendices A, B, C , and D  New data for an established program: Chapters 1 and 2, Appendices A , B, and E  Using existing data: Chap ters 1 and 3, Appendices A, B, and D  Us e in modeling: Chapters 1 and 4, Appendi x A For those familiar with EPA’s previous guidance on QAPPs Guidance for Quality Assurance Project Plans, EPA QA/G
- 5, a crosswalk between that obsolete document and this QAPP handbook is provided in Appendix E
. Questions on implementation of EPA’s Quality Management S ys tem, including development of QAPPs, may

be directed to the Quality Staff, Office of Environmental Information ( quality @ep a.gov )
.v **TABLE OF CONTENTS CHAPTER 1 INTRODUCTION**  **................................ ................................**  **................................ .**  **1** 1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 and ANSI/ASQ E4
- 2004 ................................ ................................ .......................... 1 1.2 WHAT I S A QUALITY ASSURANCE PROJECT PLAN? ................................ .. 3 1.3 THE GRADED APPROACH ................................ ................................ .................. 7 1.4 GENERI C QAPPs ................................ ................................ ................................ ... 7 1.5 PHASED QAPPs ................................ ................................ ................................ ..... 9 1.6 WHEN SHOULD A QMP BE COMBINED WITH A QAPP? .............................. 9 1.7 DEVELOPING, REVIEWI NG, AND APPROVING A QAPP ............................ 11 1.7.1 Developing a QAPP
................................ ................................ ................... 11 1.7.2 Reviewing a QAPP
................................ ................................ .................... 12 1.7.3 Approving a QAPP
................................ ................................ .................... 12 1.8 DI STRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP ................. 13 1.8.1 Distributing a QAPP
................................ ................................ .................. 13 1.8.2 Implementing a QAPP
................................ ................................ ............... 13 1.8.3 Modifying a QAPP
................................ ................................ .................... 13 1.9 PERI OD OF APPLICABILITY ................................ ................................ ............ 14 1.10 SUPERSESSI ON ................................ ................................ ................................ ... 14 **CHAPTER 2 QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT**  **MEASUREMENT ................................**  **................................ ................................**  **............ 15**  2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT MEASUREMENT ................................ ................................ ........... 15 2.2 PROJECT MANAGEMENT ( Plan )
................................ ................................ ...... 16 2.2.1 Title, Version, and Approval/Sign
- Off
................................ ...................... 16 2.2.2 Document Format and Table of Contents
................................ .................. 17 2.2.3 Distribution List
................................ ................................ ......................... 17 2.2.4 Project Organization and Schedule
................................ ............................ 18 2.2.5 Project Background, Overview, and Intended Use of Data
....................... 18 2.2.6 Data/Project Quality Objectives and Measurement Performance Criteria
................................ ................................ ................................ ....... 19 2.2.7 Special Training Requirements and Certification
................................ ...... 21 2.2.8 Documentation and Records Requirements
................................ ............... 21 2.3 DATA ACQUISIT ION (Do)
................................ ................................ ................. 22 2.3.1 Data Collection Procedure, Experimental Design, and Sampling Tasks
... 23 2.3.2 Sampling Procedures and Requirements
................................ ................... 24 2.3.3 Sample Handling, Custody Procedures, and Documentation
.................... 25 2.3.4 Analy

tical Methods Requirements and Task Description
......................... 28 2.3.5 Quality

Control Requirements
................................ ................................ ... 29 2.3.6 Instrument/Equipment Testing, Calibration and Maintenance Requirements, QA/QC for Supplies and Consumables
............................. 30 2.3.7 Data Management Requirements
................................ ............................... 32 2.4 ASSESSMENTS (Check)
................................ ................................ ...................... 32 2.4.1 Technical S ys tems Assessments ................................ ................................ 33 2.4.2 Performance Audits of Measurement and Anal yt ical Sy stems .................. 34vi 2.4.3 Surveillance of Operations
................................ ................................ ......... 35 2.4.4 Audits of Data Quality ................................ ................................ ............... 36 2.4.5 Qualitative and Quantitative Comparisons to Acceptance Criteria
........... 37 2.4.6 Interim Assessments of Data Quality ................................ ........................ 37 2.4.7 Evaluation of Unconventional Measurements
................................ ........... 38 2.4.8 Evaluation of Unconventional Monitoring Projects
................................ .. 39 2.5 REVIEW, EVALUATION OF USABILITY, and REPORTI NG REQUIREMENTS (Act)
................................ ................................ ........................ 39 2.5.1 Data Verification and Validation Targets and Methods
............................ 40 2.5.2 Quantitative and Qualitative Evaluations of Usability
.............................. 41 2.5.3 Potential Limitations on Data Interpretation
................................ .............. 42 2.5.4 Reconciliation with Project Requirements
................................ ................. 42 2.5.5 Reports to Management ................................ ................................ ............. 44 **CHAPTER 3 QAPP ELEMENTS FOR EVALUATING EXISTING DATA**  **...................... 46**  3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING DATA ................................ ................................ ................................ .................... 46 3.2 PROJECT MANAGEMENT (Plan)
................................ ................................ ...... 47 3.2.1 Title, Version, and Approval/Sign
- Off
................................ ...................... 47 3.2.2 Document Format and Table of Contents
................................ .................. 48 3.2.3 Distribution List
................................ ................................ ......................... 4 8 3.2. 4 Project Organization and Schedule
................................ ............................ 49 3.2.5 Project Background, Overview, and Intended Use of Data
....................... 49 3.2.6 Data/Project Quality Objectives and Measurement Performance Criteria
................................ ................................ ................................ ....... 50 3.2.7 Special Training Requirements and Certification
................................ ...... 51 3.2.8 Documentation and Records Requirements
................................ ............... 51 3.3 DATA ACQUISIT ION (Do)
................................ ................................ ................. 52 3.3.1 Proposed Data Source Originator and Publication Information
................ 52 3.3.2 Data Format and Accessibility
................................ ................................ ... 52 3.3.3 Establishment of Acceptance Criteria
................................ ........................ 53 3.3.4 Sample Data Collection Methodology
................................ ....................... 54 3.3.5 Quality

Program and Quality

Assurance Procedures Used b y Data Originator
................................ ................................ ................................ ... 54 3.3.6 Documentation of Sample Quality Assurance Procedures
........................ 55 3.3.7 Data Management Requirements
................................ ............................... 55 3.4 ASSESSMENTS (Check)
................................ ................................ ...................... 56 3.4.1 Qualitative Comparisons to Acceptance Criteria
................................ ....... 56 3.4.2 Quantitative Comparisons to Acceptance Criteria
................................ ..... 62 3.4.3 Assessments to Other Criteria
................................ ................................ .... 63 3.4.4 Interim Assessments of Data Quality ................................ ........................ 63 3.4.5 Evaluation of Unconventional Measurements
................................ ........... 63 3.4.6 Evaluation of Unconventional Monitoring Projects
................................ .. 64 3.5 REVIEW, EVALUATION OF USABILITY, AND REPORTI NG REQUIREMENTS (Act)
................................ ................................ ....................... 64 3.5.1 Data Verification and Validation Targets and Methods
............................ 64 3.5.2 Quantitative and Qualitative Evaluations of Usability
.............................. 64vii 3.5.3 Potential Limitations on Data Interpretation
................................ .............. 65 3.5.4 Reconciliation with Project Requirements ................................ ................. 65 3.5.5 Reports to Management ................................ ................................ ............. 67 **CHAPTER 4 QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, AND USE OF MODELS**  **................................ ................................**  **................................ ...........**  **69** 4.1 OVERVIEW OF QAPP ELEMENTS FOR MODEL S ................................ ......... 69 4.2 PROJECT MANAGEMENT ( Plan )
................................ ................................ ...... 70 4.2.1 Title, Version, and Approval/Sign
- Off
................................ ...................... 72 4.2.2 Document Format and Table of Contents
................................ .................. 72 4.2.3 Distribution List
................................ ................................ ......................... 73 4.2. 4 Project Organization and Schedule
................................ ............................ 73 4.2.5 Project Background, Overview, and Intended Use of the Model
.............. 74 4.2.6 Data/Project Quality Objectives and Acceptance Criteria
......................... 78 4.2.7 Special Training Requirements and Certification
................................ ...... 79 4.2.8 Documentation and Records Requirements
................................ ............... 80 4.3 DATA ACQUISIT ION: MODEL DEVELOPMENT, MODIFICATION, AND USE (Do)
................................ ................................ ................................ ...... 80 4.3.1 Data Requirements for Model Input ................................ .......................... 81 4.3.2 Data Management Tasks
................................ ................................ ............ 83 4.4 ASSESSMENTS: MODELEVALUATION ACTIONS (Check)
........................ 84 4.4.1 Model Evaluation Methods and Activities
................................ ................ 84 4.4.2 Evaluation of the Model
................................ ................................ ................ 85 4.4.3 Model Output Sensitivity and Uncertainty

Anal ys is
................................ . 87 4.5 REVIEW, EVALUATION OF USABILITY: MODELING RESULTS USABILITY AND REPORTING REQUIREMENTS (A ct)
................................ 87 4.5.1 Peer Review
................................ ................................ ............................... 88 4.5.2 Description of Model Documentation
................................ ....................... 89 4.5.3 Specifications for Model Maintenance and User Support
......................... 90 4.5.4 Reports to Management ................................ ................................ ............. 90 **GLOSSARY ................................**  **................................ ................................**  **................................ ..**  **91 REFERENCES**  **................................ ................................**  **................................ .............................**  **97 APPENDIX A**  **CHECKLIST OF QAPP ELEMENTS ................................**  **........................... A**  **1 APPENDIX B DATA QUALITY INDICATORS**  **................................ ................................**  **... B**  **1** B.1 Sources of Measurement Uncertainty ................................ ................................ ... B 3 B.2 E stablishing MQOs in the Context of DQOs ................................ ........................ B 3 B.3 P recision
................................ ................................ ................................ ................ B 5 B.4 B ias
................................ ................................ ................................ ....................... B 5 B.5 A ccuracy ................................ ................................ ................................ ............... B 6 B.6 R epresentati veness ................................ ................................ ................................ B 7 B.7 C omparability
................................ ................................ ................................ ....... B 9 B.8 C ompleteness
................................ ................................ ................................ ...... B 10 B.9 S ensitivity
................................ ................................ ................................ ........... B 11 B.10 The Role of DQIs in the P roject Lifecycle
................................ ......................... B 14viii

**APPENDIX C QUALITY CONTROL SAMPLES ................................**  **................................ C**  **1 APPENDIX D DATA REVIEW, VERIFICATION, AND VALIDATION FOR**  **PROJECT PURP OSES**  **................................ ................................**  **................................ ..**  **D 1**  D.1 Verification
................................ ................................ ................................ ........... D 4 D.2 Validation
................................ ................................ ................................ .............. D 5 D.2.1 Validati on Activities

– Step A: Compliance
................................ ............. D 6 D.2.2 Validation Activities

– Step B: Comparison
................................ ............. D 6 **APPENDIX E CROSSWALKS TO OTHER DOCUMENTS ................................**  **............... E**  **1** E.1 Crosswalk: Standards CLauses 7.5 to 7.10 t o Handbook QAPP and HANDBOOK QMP ................................ ................................ .............................. E 1 E.2 Crosswalk: Standards ANNEX B to QAPP Handbook
................................ ........ E 2 E.3 Crosswalk: EPA QA/G
- 5 to QAPP Handbook
................................ ..................... E 4 E.4 Crosswalk: QAPP Handbook to EPA QA/G
- 5, QA/G
- 5M, and QA/G
- 8 ............ E 6 E.5 Cros swalk: UFP
- QAPP Workbook to QAPP HANDBOOK ............................... E 9ix **ACRONYM LIST**
ANSI American National Standards Institute ASTM American Society for Testing and Materials ASQ American Society for Quality CFR Code of Federal Regulations CI O Chief Information Officer DL Detection Limit DQA Data Quality Assessment DQI Data Quality Indicator DQO Data Quality Objective EPA Environmental Protection Agency FIFRA Federal Insecticide, Fungicide, and Rodenticide Act FO RMS Field Operations and R ecords Management Sy stem GPS Global Positioning System ICP Inductively

Coupled Plasma IQ G In formation Quality Guidelines ISO International Organization for Standardization (not an acronym) IT In formation Technology LOD Limit of Detection LOQ Limit of Quantitation MCL Maximum Contaminant Level MDL Method Detection Limit MPC Measurement Performance Criteria MQO Measurement Quality Objective OEI Office of Environmental Information OMB Office of Management and Budget ORD Office of Research and Development OSWER Office of Solid Waste and Emergency Response PCB Polychlorinated biphenyl s PE Performance Evaluation PQL Practical Quantitation Limit PQO Project Quality Objectives PRP Potentially

Responsible Parties PT Proficiency

Testx

QA Q uality

Assurance QAPP Quality

Assurance Project Plan QC Quality

Control QL Quantitation Limit QMP Quality

Management Plan QMS Quality

Management S ys tem RCRA Resource Conservation and Recovery Act SOP Standard Operating Procedure TSA Technical S ys tems Asse ssment TSCA Toxic Substances Control Act UFP Uniform Federal Policy XRF X - ray

FluorescenceQAPP Handbook 1 December 2012 Agency Review Draft **CHAPTER 1 INTRODUCTION**  **1. 1**  **QAPP s**  **, EPA QUALITY MANAGEMENT S**  **YSTEM , EPA**  **POLICY 2106 AND**  **ANSI/ASQ E4 -**  **2004** This Chapter lays a general foundation for a QAPP and provides a more detailed explanation of what is required in the Internal and External Standards. The Chapter discusses key aspects of what a QAPP contains, what it applies to, its connection to a QMP, a nd who is responsible for developing one. It lays out the general alignment of required content in a Plan
-Do
- Check
- Act paradigm. This document presents non
- mandatory guidance intended to help its users prepare Quality Assurance Project Plans (QAPPs) that meet the requirements of the follo wing two Environmental Protection Agency ( EPA ) Standards :
 Quality Standard for Environmental Data Collection, Production, and Use by EPA Organizations, (CI O 2106 - S - 01), “Internal Standard” ( EPA 2013a) ; and  Quality Standard for Environmental Data Collection, Production, and Use by Non
-EPA (E xternal) Organizations, (CIO 2106
- S - 02) , “External Standard” ( EPA 2013b )
. These Standards provide the foundation for the Agency
- wide Quality Program for environmental data
- related product s and services that are disseminated outside the Agency. They conform to EPA Quality Policy , CIO 2106.0, “ Quality Policy ” ( EPA 2008a) and to EPA Procedure for Quality Policy, CI O 2106 - P - 01.0 , “Quality Procedure” ( EPA 2008b ) as issued by the Chief In formati on Officer (CIO), EPA. The use of this Handbook is discretionary but should be used in conjunction with the two EPA Quality Standards. Throughout Chapters 2, 3, and 4 of this document, the corresponding sections of the Internal and External Standards are noted in italics
. This handbook follows the lead of the Standards in reaffirming the applicability of the national consensus standard , Quality System s for Environmental Data and Technology Programs : Requirements with Guidance for Use , ANSI/ASQ E4
- 2004 (AN SI2004 ) developed by the American National Standards Institute (ANSI) and the American Society for Quality

(ASQ)
. The QAPP document Requirements for Quality Assurance Project Plans, EPA QA/R
- 5 (EPA 2001a) , has been withdrawn in favor of these Internal an d External Standards. The QA/R
- 5 document should not be used for developing QAPPs except where explicitly allowed b y the Standards. The guidance that accompanied QA/R
- 5, Guidance for Quality Assurance Project Plans, EPA QA/G
- 5 , has also been withdrawn as i t is superseded b y this document , EPA Guidance on Quality Assurance Project Plans ( CIO 2106 - G - 05 QAPP ) , “QAPP Guidance” . It is important to remember that unlike the Standards, which present normative requirements, this document consists of informative (non
- mandatory) guidance presented in non
- prescriptive language . It is offered as a helpful resource for those who need to prepare QAPPs that comply with EPA Standards. Itmay be helpful to those creating work plans, sampling and anal ys is plans, field impl ementation plans, or other project planning documents.QAPP Handbook 2 December 2012 Agency Review Draft EPA also allows use of the Uniform Federal Policy for Quality Assurance Project Plans, ( EPA 505 - B - 04 - 900A), “UFP - QAPP ” ( EPA 2005a) as an alternative approach to satisfy EPA’s requirement for a QAPP if allowed b y the external agreement . In 2005, the Intergovernmental Data Quality Task Force (IDQTF) issued the UFP
- QAPP to address environmental data collection, production, and use for hazardous waste cleanup activities at federal facilities and installati ons 1 . Shortly

thereafter, EPA’s Office of Solid Waste and Emergency Response issued a directive (OSWER Directive 9272.0
- 17) ( EPA 2005c ) stating that the UFP
- QAPP format should be used for EPA
- managed Federal Facility projects where environmental data are c ollected and used. Both that directive and further guidance issued jointly by EPA’s Federal Facilities and Restoration Reuse Office and Quality Staff, Office of Environmental Information (OSWER 9272.0 - 20) ( EPA 2005d) state that QAPPs prepared and approved according to the UFP will be considered consistent with EPA Standards . The UFP
- QAPP worksheets are also consistent with the EPA Quality Standards. Figure 1 depicts the hierarchical relationship of QA documents : ANSI/ASQ E4
- 2004 national consensus standard , together with the Information Quality Act of 2001 ( IQ G 2001 ) , establish a basis for the Agency ’s Quality Policy ; the Quality Procedure provides additional explanation about how to carry

out the Policy ; the two Standards contain requirements for applying the Policy and Procedure to environmental data operations (within and external to the Agency); Handbook for Preparing Quality Management Plans (EPA 2013c ) offers help in preparing Quality Management Plans (QMPs) that meet the re quirements of the higher level documents; and this Handbook document offers guidelines, advice, and examples that will help users satisfy provisions of the Standards at the project level
.

**Figure 1. R** **elationship among**  **EPA Quality Program**  **Document s** 

1 The IDQTF consists of representatives from EPA , the Department of Defense (DoD), and the Department of Energy (DOE). It was established to address real and perceived inconsistencies and deficiencies in quality control for laboratory data, within and across governmental organizations, which result in greater costs, time delays, and increased risk. It is chaired by the Director of EPA’s Federal Facilities Restoration and Reuse Office and operates as a partnership, reaching decisions through consensus. **Handbook for Preparing Quality Management Plans**  **Quali ty Procedure** 

**CIO Procedure 2106 -**  **P -**  **01 .0** 

**Handbook for Developing Quality**  **Assura nce Project Plans**  **Internal Standard CIO 2106**  **- S**  **- 01**  **.0 External Standard**  **CIO 2106**  **- S**  **- 02**  **.0 Quality Policy CIO 2106**  **National Consensus Standard**  **ANSI/ASQ E4 -** 

**2004 Information Quality Act** QAPP Handbook 3 December 2012 Agency Review Draft **1.2 WHAT IS A QUALITY AS**  **SURANCE PROJECT PLAN ?**  This section describes QAPPs in a Question
- and
- Answer format and explains more fully the requirements of Standard 7. 5, Quality Assurance Project Plan, and Annex B 1 , Introduction, B2, QAPP Responsibilities and Approvals, B2.3, Applicability, and B3.2, General Content Requirements
. A QAPP is a formal document describing in comprehensive detail the necessary quality assurance (QA), quality control (QC), and other technical activities that will be implemented to ensure that the results of the work perform ed will satisfy the stated performance criteria
. It describes the activities of a project in the acquisition of environmental data or information from direct measurement activities, existing data, or generated b y models. What is the difference between QA a nd QC? Q A refers to the s ys tem of management activities that are designed to ensure quality , whereas QC refers to the sy stem of preventive and corrective technical activities that measure or monitor performance against defined standards and makes adjustments to maintain acceptable performance. Wh y is a QAPP important? A QAPP presents the steps that should be taken to ensure that environmental data collected , produced, and used are of adequate and sufficient ty pe and quality required for a specific decision or use . It presents an organized and s ys tematic description of the way s in which QA and QC will be applied to the collection and use of environmental data
. It documents the results of the project’s planning process thus providing a concise record of the c onduct of environmental data operations. Must I have a QAPP? ANSI/ASQ E4
- 2004 Section 6 (Part B) (ANSI 2004) and the EPA Internal and External Standards , Clause 7 (EPA 2013a , EPA 2013b ) require that a QAPP be approved for all data collection projects
. The specific requirements for the content of a QAPP are presented in Annex B of each Standard. What are the benefits of a QAPP? A QAPP (or equivalent document as stipulated in the organization’s QMP) integrates QA and QC a spects of a project throughout its lif e cycle, including planning, implementation, assessment, and corrective actions
. A QAPP is a “roadmap” for ensuring the implementation of quality procedures and practices in a project
. When impl ement ed as prescribed , a QAPP increases efficiency and enable s decisions to be made effectivel y with greater certainty and reduces the potential for possible rework
. To what does a QAPP apply ? The ultimate success of an environmental program or project depends on the quality of the environmental data collected and used in decision
- making, and this may depend significantl y on the adequacy of the QAPP and its effective implementation. 2 The title of the QAPP being developed should include information as to what the QAPP will apply to and bear a unique identifying number (or QAPP Tracking Number) such that it can be readil y identified amongst similar QAPPs.

2 This paragraph is taken verbatim from the introduction to the Workbook for Uniform Federal Policy for Quality Assurance Project Plans (EPA
- 505
-B
- 04
- 900C) (EPA 2005b) . Throughout the QAPP Guidance there are several instances of shared language with the UFP
- QAPP. This serves as a general notice that these documents are consistent by design, and specific citations will not be provided.QAPP Handbook 4 December 2012 Agency Review Draft Agency activities involving environmental data required by the Internal and External Standard s to have a QAPP include, but are not limited to:

 direct and indirect field and/or laboratory measurements ;  evaluating the operation and performance of environmental technology ;  inspections;  existing environmental data;  questionnaire survey

development or application;  development and validation of sampling or anal ytical methods ;  environmental model modification and/or development;  enforc ement monitoring or assessments;  application of environmental mana gement s ys tems;  environmental safety

and health monitoring;  scientific research;  regulatory

development;  statistical or ec onomic analy ses of environmental data;  use of information technology (mathematical models) ;  use of information sources outside of direct EPA management controls or authority; and  use of o ther data sources (e.g., literature or the Internet). In some instances an organization may develop a QAPP for field collection activities and another for laboratory

anal ys is activities; a clear distinction between them is advisable. For projects with multiple participants each developing a QAPP, an overarching QAPP which would provid e requirements such that individual components may be effectivel y integrated is advised. Who is involved in developing a QAPP? Most activities involving environmental data require the coordinated efforts of many individuals, possibly including managers, engineers, scientists, statisticians, information technology (I T) experts, modelers, and others
. The QAPP should integrate the contributions and requirements of everyone involved into a clear, concise statement of what needs to be accomplished, how it will be done, and b y whom
. It should provide understandable instructions to those who must approve or implement the QAPP, including the field sampling team, the anal ytical laboratory , and data users and reviewers
. Be yond the general guidanc e provided in the QAPP, it should identify and integrate the use of related policies and procedures applicable to the project such as administrative procedures, laboratory procedures, data anal ys is methodologies, ITpolicies and procedures, and data handli ng and anal ys is policies. How is the QAPP effective? The QAPP must specify the level or degree of QA and QC activities needed for the particular environmental data or model operations with clear objectives, acceptance criteria, and QA/QC control strategies
. The QA and QC technical requirements of a project should be commensurate with the type of work to be done (e.g., monitoring, siteQAPP Handbook 5 December 2012 Agency Review Draft characterization, model simulation, and bench level proof of concept) , the purpose of the project (e.g., regulatory

enforce m ent, development work for rulemaking , permit approval, research publications , and journal articles) , and the scale of the project such as a one - time assessment or a template for multiple assessments
. Regardless of the complexity of the project, the QAPP do cuments how the project team will ensure that the quality of data is suitable for its intended use by documenting acceptance criteria against which assessments may be made
. Wh y is it important to me? The QAPP is invaluable in not only documenting all aspec ts of the project serving as a structured resource for writing the final report on the project. In addition, i t helps a successful implementation during reassignments of personnel. It is the basis for assuring the integrity of the project’s conclusions. Who approves a QAPP before being put into operation? Each QAPP must be approved b y the organization’s project manager or director and the QA Manager prior to any data collection, production, or use (with the possible exception of circumstances requiring im mediate action to protect human health and the environment, or operations conducted under police powers). If this approval function has been delegated, it must be in accordance with the organization’s QMP. How do I develop a QAPP? A brief summary of the pr ocess: 1. determine the scope and objectives of the project and gather background information; 2. assemble a project team with necessary project and QA expertise; 3. conduct planning sessions to determine how t o collect the specific typ e of data , the amount of data , and what the goals or acceptance criteria will be; 4. using the Standard and Handbook (if necessary ) develop a draft QAPP; 5. circulate the draft QAPP for peer review, comment, and input for improvement; 6. submit the final QAPP for approval; 7. after approval , distribute the QAPP to all persons involved with the project; and 8. proceed to implement the QAPP allowing for documentation of changes, re
- approvals, and distribution of updated QAPPs. How is a QAPP organized? The QAPP elements are arranged in four sections to reflect the l ogical flow of the project life cycle (also known as the Plan
- Do - Check
- Act steps , Figure 2 ).

**Figure 2. Plan -**  **Do -**  **Check -** **Act Project Life Cycle**QAPP Handbook 6 December 2012 Agency Review Draft The four ty

pes of elements in these steps are:
 Project Management ( Plan )
- These elements , coverin g the basic area s of project management, project objectives, and the roles and responsibilities of the participants , document th e development of progress towards the project ’s goal . The development of technical and quality project objectives , such as Data Quality Objectives (DQOs), are part of this step
.  Data Acquisition (

Do )
- These elements cover all aspects of data measurement , data acquisition design s, and their implementation
. The development of performance and acceptance criteria for the collection of data by direct measurement, acquisition from existing sources, and b y modeling are integral to this step. This ensures the intended measurements , data collection , or acquisition methods are appropriate for achieving project objectives
.  Assessments ( Check

)
- These elements address the activities for assessing the effectiveness of the implementation of the project and associated QA and QC activities
. The purpose of assessment is to ensure that the QAPP is implemented as prescribed and that project acti ons are implemented as expected.
 Review, Evaluation of Usability, and Reporting Requirement s (A c t)
- These elements cover the QA activities that occur after the data collection (or use) phase of the project is completed
. Implementation of these elements w ill help in determining that the data conform to the specified criteria, and ensuring that data usability can be documented in a final project report. This step includes any limitations or restrictions on the use of the data, or contingencies for revising project objectives. How is this QAPP Handbook organized? This document covers QAPP development for three different types of projects:  studies in which new data will be collected (Chapter 2);  investigations in which existing data will be used (Chapter 3); and  development, modification, and use of models using environmental data (Chapter 4). These three project ty

pes are each addressed in their own chapter and should be used with the relevant a ppendices of this document. It is not the intent of this Handbook to specificall y identify every type of project for which a QAPP will be needed, but rather to describe QAPP elements for general types of projects. If a QAPP element is not applicable, then an indication of why it is not relevant should be given
. If information is available elsewhere should I rewrite this into the QAPP? Reference to existing documents (such as standard operating procedures, sampling and anal ys is plans, environmental assessments, work plans, literature files, full y documented models) can greatly reduce the size of a QAPP. Documents may be included by

reference or appended to the QAPP. It is essential, however, that everyone , including QAPP reviewers, involved with the project has access to the referenced documents.QAPP Handbook 7 December 2012 Agency Review Draft How long is a QAPP? The QAPP needs to have enough information to adequately describe the project’s objectives, implantation, and results. This implies a complex project will demand more than a relativel y simple study . The use of a “graded approach” is recommended. Is guidance written by other organization based on EPA QA/G
- 5 obsolete? All of the strengths of the now obsolete guidance EPA QA/G
- 5 (EPA 2002a) have been incorporated into this QAPP guidance. This means that guidance written b y other organizations based on the obsolete document will still be very strong, but will not have some of the latest advances in QA practices
. O rganizations are encouraged to review their guidance to ensure it meets the requirements of the Standards, and also take advantage of some of the advice offered in this QAPP guidance. **1.3 THE GRADED APPROACH**  The graded approach is an extremely important concept and the outline given in Standards Annex B3.1, General Requirements, amplified by dis cussion
. Recognizing that a “one size fits all” approach to quality requirements will not work in organizations as diverse as EPA, implementation of the EPA Quality Program is based on the principle of a graded approach
. The graded approach is the process of establishing the project requirements and level of effort commensurate with the intended use of the results and the degree of confidence needed in the quality of the results. The graded approach allows for the content and level of detail in each QAPP to vary according ly
. For exampl e, the quality expectations of fundamental research may be different from that of regulatory

compliance because the purpose or intended use of the data is different
. The intent of the graded approac h is to achieve sufficient detail in the QAPP to satisfy the objectives of the project , ensure adequate quality for the intended use of the data , and meet the resources allocated to the project
. Cooperation and discussion during the QAPP development process is encouraged, especiall y on the implementation of the graded approach
. The final decision on QAPP content and level of detail belongs to the EPA organization responsible for the work to be done, c onsistent with the approved QMP , and within the resources available for the project
. **1. 4**  **GENERIC QAPP s**  Many organizations are engaged in projects or programs collecting data under very similar circumstances. Th is section offers advice on the use of generic QAPPs using the requirements of Standards Clause 7.5, QAPP, and Annex B3, General Requirements
. A generic QAPP provides an overarching plan that describes the quality objectives and documents a comprehensive set of sampling, analy sis, QA/QC, data review, and assessment procedures specific to a large program or long
- term project
. In contrast to the project specific QAPP, the generic QAPP serves as an umbrella under which multiple data collection, production and use activities may

be conducted over an extended period of time
. Generic QAPPs may

make sense in situations such as where multiple sites , sy stems , or projects will be sampled under a common sampling and analysis protocol/plan
. A simple way to determine whether a generic QAPP is appropriate for a project is to consider whether there isQAPP Handbook 8 December 2012 Agency Review Draft sufficient consistency

across the QA needs of multiple projects within a program that combining the planning into a single , generic QAPP will:
 ensure the necessary

level of quality

for all projects covered b y the generic QAPP; and  require less time and resources to manage with a single generic QAPP than with multiple project specific QAPPs. An approved generic QAPP should be supported by task or project specific addenda , which address the issues unique to each task or project. Project or task specific information that is not covered b y the generic Q APP should be documented in detail in these addenda . The generic QAPP should specify

the preparation, review, and approval of task or project specific addenda. For external non
- EPA projects, EPA may authorize the organization to approve project specific ad denda contingent upon a review and approval process that is fully documented in the approved generic QAPP. Some organizations may

find a variant on generic QAPPs, developed by EPA Region 9, more suitable to their needs. Called Region 9 QA Program Plans , “ ... they are intended to establish policies that define and document the typ e and quantity of data needed for program level environmental decisions and to describe the methods required for collecting, anal yz ing, and assessing data to support those decisions. For the purposes of this [EPA Region 9] guidance, an environmental program is considered to be a series of activities which are based directly or indirectly on an act of Congress and defined in regulations promulgated b y EPA, State, or tribal governments...The measurements under a program generally reflect on
- going activities wh ich do not have defined start and end dates (not to be confused with grant cycles). This does not preclude a program implementing or conducting projects. Program activities are usually of a recurring nature although specific activities may

not recur. For example, there may be on
- going water monitoring, but sampling locations may change from year to year. EPA and/or a funded organization may sponsor environmental projects. Projects are considered to be activities that have a definable beginning and a definable end. Projects would be expected to have their own QAPP s that would either be approved by EPA or by

the other organization. A QA Program Plan should describe policies regarding what ty

pe of grants, cooperative agreements, or activities require a QAPP or other QA documentation, what the nature of that documentation would be, and what activities might be exempted from that requirement. For example, an investigation of an integrated p est management approach under FIFRA or an evaluation of a new wastewater treatment methodology would be considered projects that included environmental measurements that would be under a broader environmental program. A QA Program Plan should also describe what information the QAPP should include or cite appropriate guidance...Both QA Program Plans and QAPP s should include specific identifiable goals and objectives and discuss the uses of the data. Often, but not alway s, program goals are more closel y tied to environmental regulations, whereas project goals are developed for that event based on a sy stematic planning ” ( EPA 2012 ).QAPP Handbook 9 December 2012 Agency Review Draft **1. 5**  **PHASED QAPP**  **s** Some organizations need QAPPs that are brought on
- line as a project is being developed. Th is section offers advice on the use of a set of QAPPs using Standards 7.5, QAPP. When there is significant uncertainty surrounding a project and additional clarity is expected as initial data or information are gathered, or for a project that is iterative in nature, it may be appropriate to use a phased approach to QAPP development
. The phased QAPP should inc lude a description of the decision points for the project. Examples of when a phased QAPP might be appropriate include:
 literature review s in which the eventual directio n and depth of the research is dependent upon the information found in the articles in the first round of the project;  investigation s or phased project cycle s where the results of the initial assessment will dictate the following steps (e.g., no contaminants found above thresholds leading to site closure vs. contaminants found above thresholds leading to a risk assessment and feasibility study);  anal ytical method development for a specific purpose (e.g., improved sensitivity) for which an earl y step in the process may determine the viability

of continued effort ; or  model development and implementation for which the project overview, objectives, and measurement performance criteria can be defined upfront, but until the model requirements are defined, the requirements for input data can’t be clearl y established. If a phased QAPP is developed, it should still contain all relevant elements of a QAPP
. The efficiency to be gained from a phased QAPP may be realized by

not spending time or resources to specify the de tailed project requirements for each “what
- if” scenario for the project
. If a phased QAPP is developed, it is very important that it be updated as the direction of the project becomes clear
. No project work should take place without an appropriate QAPP in place, so moving beyond the phase documented in the QAPP should not occur until the QAPP is updated
. These updates should be described in the revision history (see Section 1.8)
. **1.6 WHEN SHO**  **ULD A QMP BE COMBINE D WITH A QAPP?**  Some organizations have no direct need for a full Quality Management Plan (QMP) but could incorporate needed elements into a combined QMP/QAPP outlined in Standards Clause 7. 5.2. The role of a QMP is to define and describe an organization’s overall structure, processes, roles , and responsibilities for planning, implementing, and evaluating environmental data operations
. The role of a QAPP is to provide a road map for environmental data production, collection, and use in a project
. QAPPs document project objectives along with the QA and QC procedures that will ensure that those objectives are achieved
. While the QMP defines the Quality Program at the organizational level, the QAPP defines project
- specific quality concerns . There may be, and usually are, multiple QAPPs developed under a QMP
. Figure 3 depicts the relationship between QMPs and QAPPs within an organization’s Quality Management Sy stem ( QMS ). QAPPs are needed for projects where collection of new data is planned, e xisting data will be incorporated to meet project objectives, or models will be used to evaluate and interpret data.QAPP Handbook 10 December 2012 Agency Review Draft **Figure 3 .**  **Relationship between**  **QMP and QAPPs** In some circumstances, EPA may recommend the preparation of a combined QMP and QAPP as an appropriate means to document organizational level and project
- level environmental data activities
. The goal of the project, together with its data needs, is the principal driver of the QA/QC activities documented in the combined QMP/AQPP. Ex amples where this may be appropriate could include research projects, grants, inspection programs, and monitoring networks involving mul tiple small organizations
. In such cases, a combined QMP and QAPP can be an effective means to document organizational level and proje ct - level environmental data activities, particularl y when a f ull QMP is not needed
. Combined QMP/QAPPs should not be used for external agreements containing multiple projects that may be determined over the life of the agreement such as mission contracts and program grants)
. Some specific situations in which a combined QMP/QAPP might be appropriate include (but are not limited to):
 routine inspection programs where inspections are conducted in the same manner for all visits (e.g., EPA lead or radon program s);  small organizations that do infrequent work with EPA funds (e.g., competitively awarded one - time grants and environmental education grants);  one - time mobilization to a previousl y unvisited field location such as for monitoring assistance to a Tribal lo cation;  necessary

quick
- turnaround data anal ys e s that may involve new laboratory facilities, methods, or equipment;  environmental measurements related to technology performance, where QMS planning for the operation of the technology is already

addressed by another document;  academic research activities of a one
- time nature where an organizational QMP has more detail than needed; or  a monitoring program conducted b y a volunteer organization where the size and structure of the organization is limited. A combined QMP/QAPP incorporates some elements of the QMP and some elements of the QAPP . A combined QMP/QAPP is primarily a QAPP; that is, it pertains to a specific project orQAPP Handbook 11 December 2012 Agency Review Draft program . However, the project may not require the application of a full QMS
. As wi th any QMP, it should document the management controls that should be applied to the planning, implementation, and assessment of quality activities, and what processes will be followed to carry out those activities
. These may vary depending upon the scope and objectives of the project and should be defined b y the EPA organization. It is EPA’s responsibility to ensure that all elements of a combined QMP/QAPP , as defined b y the EPA organization, are addressed b y the organization performing the work
. The combi ned QMP/QAPP is subject to the same EPA review and approval as any other QAPP
. The QMP elements of the combined QMP/QAPP should be subject to appropriate QMP
- level review. Further guidance for a combined QMP/QAPP is provided in the QMP Guidance
. **1.7 DEVELOPING, REVIEWIN**  **G, AND APPROVING A Q APP**  Th is section offers guidance on Standards Clause 7.5 QAPP , and Annex B 2.1 , QAPP Preparation Responsibilities and App rovals, and Annex B2.2 QAPP Implementation and Revision
. 1.7.1 Developing a QAPP The QAPP may

be prepared by

an EPA organization or a non
- EPA organization (e.g., a contractor, an assistance agreement holder, another Federal agency under an interagency agreement)
. All EPA organizations collecting, producing, or using environmental data must have their QAPPs approved by their managers and QA Manager or designee as specified in the organization’s QMP. Except where specifically delegated in the QMP of the EPA organization sponsoring the work, all QAPPs prepared b y non - EPA organizations must be approved b y EPA before implementation, and involvement of EPA and other regulatory agencies, as applicable, in development is encouraged. QAPP development necessitates the coordinated efforts of many individuals such as those who will gather data and those who will use the data to make decisions
. These individuals include: decision makers, project managers, regulators, stakeholders, and technical staff (e.g., hydrologists, biologists, chemists, samplers, statisticians, engineers, economists, modelers, and risk assessors)
. In addition, peer reviewers and individuals with varied expertise ensure that technical areas are sufficiently addressed, thus helping to identify important issues during review, approval, and implementation. For internal EPA projects, th e Project Manager (PM) or Principal Investigator (PI) is generally responsible for overseeing the preparation of the QAPP
. For external projects, the funded entity is usually responsible for QAPP development ( grantees being allowed to develop an independent QAPP) , but for other agreements (for example, contractual) the QAPP is subject to review and approval b y EPA prior to implementation
. Inthe case of another Federal agency receiving funds from EPA, as through an interagency agreement, the award indicates who is responsible for QAPP preparation
. When EPA receives project funds from another Federal agency , EPA personnel usuall y write the QAPP unless the other organization negotiates specific quality requirements. It is usually advantageous to hold round
- tab le discussions between stake
- holders , QA specialists, and Project Manager prior to writing the QAPP. This ensures the coverage of the QAPP is inclusive and has a broad coverage of the project
.QAPP Handbook 12 December 2012 Agency Review Draft Chapters and appendices of greatest utility when creating a QAP P vary with the intent and purpose of the data collection:
 New data for a new project: Chapters 1 and 2, Appendices A, B, C, and D  New data for an established program: Chapters 1 and 2, Appendices A , B, and E  Using existing data: Chapters 1 and 3, Appendi ces A, B, and D  Use in modeling: Chapters 1 and 4, Appendix A 1.7.2 Reviewing a QAPP A QAPP can be reviewed by several authorized EPA reviewer s to ensure it contains the appropriate content and level of detail as defined by the EPA organization’s QMP
. These reviewers may

include the Project Manager, Contracting Officer’s Technical Repre sentative, and QA Manager . In some cases, the authority to review and approve QAPPs is delegated to another part of the EPA organization covered b y the same QMP
. Incases where the authority to review and approve QAPPs is delegated in writing b y EPA to another organization (e.g ., a Federal agency or a State under an EPA
- approved QMP when the environmental data operation has been delegated to that organization for implement ation), it is advised that the EPA P roject M anager and EPA QA Manager should be involved in the review and approval steps. Reviewers with expertise in the project specific areas , such as program managers (decision makers), QA staff independent of project management, and field and laboratory technical staff, should review the plan
. These reviewers should:
 ensure that the information is accurate and complete;  ensure that all appropriate elements are addressed;  ensure that the plan identifies the project's te chnical and quality objectives, and that the intended measurement and data acquisition methods will satisfy these objectives;  c onfirm that the planned assessment procedures will be adequate to evaluate the project; and  c onfirm that there is a process to identify any limitations on the use of the data. 1.7.3 Approving a QAPP The EPA organization's QMP establishes how, when, and b y whom development, review, approval, and effective oversight of QAPPs should occur
. This includes pr ocesses for external (non
- EPA) organizations that prepare QAPPs
. For EPA projects, the Project Manager ( or Project Officer in the case of external funding) , and the QA Manager usuall y approve the QAPP
. For external non
- EPA projects, the responsible organiz ation’s Project Manager or Principal Investigator and QA Manager may review and approve the QAPP and then submit it for EPA approval (unless that EPA organization has specifically delegated approval in its Agency
approved QMP)
. It is also beneficial if oth er key staff such as the laboratory or division directors , and prime contractors and subcontractors, sign the plan to indicate their review and approval.QAPP Handbook 13 December 2012 Agency Review Draft No work involving environmental data collection, production (or generation), or use shall be started until the QAPP has been approved b y the EPA QA Manager (or authorized representative). Some organizations, according to their QMP, may permit conditional approval of a QAPP in situations where only

non - critical deficiencies are unresolved (e.g., missing a final organizational chart)
. Conditional approval allows project work to start while the QAPP is finalized
. The QAPP is then resubmitted for approval when the remaining deficiencies are resolved. Any work performed under conditional approval should be in a ccordance with the fully approved QAPP , with any deviations being full y documented for resolution
. **1.8 DISTRIBUTING, IMPLEM**  **ENTING, AND MODIFYIN G A QAPP**  Th is section offers guidance on Standards Annex B 2.2, QA Implementation and Revision
. 1.8.1 Distributi ng a QAPP All applicable personnel involved in the project should retain or have access to the current version of the QAPP
. This may include the Project Manager, QA Manager, modeler, data reviewer, laboratory

manager, field team leader, and any essential c ontractor and subcontractor personnel involved with the project. 1.8.2 Implementing a QAPP The organization performing the work is responsible for ensuring that the QAPP is implemented as written and approved, whether this work is conducted b y EPA or non - E PA personnel
. Ultimately

the Project Manager is responsible for project activities
. A clearly written QAPP will help the Project Manager implement the plan, because all project personnel will understand the specifications before the start of data production or data use activities. However, effort is needed throughout the entire project lifecycle to ensure by direct evidence that the QAPP is being implemented properl y as planned. 1.8.3 Modifying a QAPP The QAPP’s effective period of approval will b e defined b y the parent EPA organization that approves subordinate EPA organizations and non
- EPA organization QAPPs
. Ingeneral, the QAPP approval should encompass the project
- specific activities and proposed timeframe to complete the activities
. A QAPP ap proval for an external agreement may not exceed the period of performance for the agreement or no - cost extensions to the period of performance
. The QAPP should be kept current and all appropriate personnel involved in the work should have easy access to a current version of the QAPP
. Although the approved QAPP must be implemented as prescribed, it is not inflexible
. Due to the complex and diverse nature of environmental data operations, changes to original plans are often needed
. When such changes occur, th e organization’s QA Manager (or designee) , together with the Project Manager, determines if the change significantl y impacts the technical and quality objectives of the project
. When a substantive change is warranted, the originator of the QAPP modifies th e QAPP to document the change and submits the revision for approval b y the same authorities that performed the original review
. The EPA Project Manager should determine whether the need toQAPP Handbook 14 December 2012 Agency Review Draft revise the QAPP merits stopping project activities until the revise d QAPP is approved (e.g., if serious safety threats are identified that haven’t been addressed in the original QAPP), or if the ty pe of revisions required do not necessitate stopping work (e.g., the data validation firm named in the QAPP went out of busine ss and an equally competent firm was identified to fill that role)
. All revisions should be documented in the QAPP revision history
. For programs or projects of long duration such as multi
- year monitoring programs, the QAPP should be reviewed regularly as specified in the organization’s QMP. If it is necessary to modify the QAPP during the timeframe when it is in effect, the same steps should be taken as in the initial phase to ensure that the revised QAPP is reviewed and approved b y the original approvers or by appropriate personnel as specified in the organization’s QMP
. **1.9 PERIOD OF APPLICABILITY**  Th is section offers guidance on Standards Annex B 2.3, Applicability of QAPPs. Consistent with EPA policy , this Handbook is valid for five ye ars from publication and will be subject to review at which time it will be revised and reissued, reaffirmed without change, or withdrawn. If circumstances warrant, this handbook may also be revised during the course of the five
- ye ar period. **1.1 0**  **SU PERSESSION**  This document conforms to the current editions of Quality Policy CI O 2106, and Quality Procedure 2106 - P - 01.0, as well as the Internal and External Standards. This document supersedes the previous QAPP
- related guidance documents EPA QA/G
- 5 (EPA 2 002a) and EPA QA/G
- 5M (EPA 2002c).QAPP Handbook 15 December 2012 Agency Review Draft **C HAPTER**  **2 QAPP ELEMENTS FOR THE**  **COLLECTION OF DATA BY DIRECT MEASUREMENT** The collection of data by direct measurement describes the activities where the Agency, its contractor, or organization working in agreement with Agency, gathers new data for a project. The internal Standard applies to Agency activities, the External Stand ard to non
-EPA organizations. The two Standards run in parallel with the exception of Internal Clause 7.11 Management of Quality Requirements for External Agreements, which applies only to Agency personnel involved with organizations having agreements wit h the Agency. **2 .1**  **OVERVIEW OF QAPP ELE MENTS FOR THE COLLEC**  **TION OF DATA BY DIRECT MEASUREMENT**  The QAPP integrates all technical and quality aspects for the life cycle of the project including planning, implementation, and assessment
. The ultimate success of an environmental program or project depends on the quality of the environmental data collected , produced, and used in decision
- making, and this quality depends significantly

on the adequacy

of the QAPP and on its effective implementation. Probably the mos t important part of the QAPP is in the planning for obtaining the data or information because an error at this stage may be difficult to rectify

once the project is underway . One of the key outputs of the sy stematic planning process is the construction of a conceptual model that describes the scientific and engineering process under investigation. The conceptual model is an important tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. Further details on sy stematic planning are to be found in Annex C of the Standards and EPA 2006a. The format of the QAPP is not critical except that all required elements must be addressed and the content of the QAPP presented in a logical, easily understood manner. In some cases, certain elements will not be appropriate for a particular project
. Elements that do not apply can be addressed with a simple statement of wh y the information is not relevant or with a cross
reference to another approved document in which the information appears
. It is acceptable to add sections to a QAPP beyond those identified herein
. For example, if a project includes the temporary

care of live animals such as laboratory mice , additional sections relevant to activities involving the care and feeding of the animals could be included. In addition , any other applicable policies or procedures (e.g., administrative, IT, or long
- term data management) may be referenced or included as beneficial for the quality of the project
. I f use of a particular information sy stem or hardware/software configuration is vital to project success, the QAPP should address IT quality considerations, including a l isting of IT
- related procedures that will be applied
. EPA organizations should consider the set of CI O policies, procedures, standards, and guidelines for this purpose
. For n on - EPA organizations , the applicable external agreement may incorporate relevant I T guidelines. Documentation, such as an approved Work Plan, SOPs, model calibration reports, etc., may be referenced in response to a particular required QAPP element to reduce the size of the QAPP and the time required for preparation and review. All referenced documents should be attached asQAPP Handbook 16 December 2012 Agency Review Draft appendices to the QAPP itself or be placed on file with the appropriate EPA organization and available for routine referencing when needed. This chapter focuses on QAPPs for projects that involve the generation of new data
. The el ements are arranged according to the Plan
- Do - Check
- Act project life cycle. **2 .2**  **PROJECT MANAGEMENT (**  **P LAN**  **)** Planning, th e first part of the construction of a QAPP , deals mainly with the management aspects of the project. Most of the structure is simple good management practices, the only QA
related part being the establishment of data or project quality objectives and measurement performance criteria (section 2.2. 6 ). The section offers guidance on Standards Clause 7.5, QAPP, Standards Clause 7.7, Systematic Planning, Annex B3.3 Project Management, and Annex B3.4 Data Acquisition. The elements in this section address the format and disposition of the QAPP, project a dministrative functions , project information and goals
. These elements document the backbone of the project planning process and lay the groundwork for the more technical elements. The QAPP must describe the project adequatel y and the elements that can address the basic project management and objectives of the work include:
 title, version number, an d approval/sign
- off sheets;  document format and table of contents;  distribution list;  project organization and schedule;  project/problem background and description;  data or project quality objectives and measurement performance criteria;  special training requirements/certification; and  documentation and records requirements. The records for a QAPP should conform to the EPA Records Management Policy (EPA 2009a). 2.2.1 Title, Version, and Approval/Sign
- Off Each QAPP should include a page with the title of the project and the name of the organizations involved in various aspects of that project . The version and number assigned to the QAPP should also be clearl y identified along with the title. It is acceptable to create separate title pages and signature pages, as long as the document title, version number, and date appear on the signature page. The names, titles, signatures, and signature dates of those approving the plan are also placed on this page. Individuals responsible for approving the QAPP may include the organization’s Technical Project Manager and QA Manager, and the EPA (or other funding agency) Project Manager and QA Manager
. Their signatures indicate both their approval of the plan and commitment to follow the procedures noted. Other ke y personnel that may sign the plan are the laboratory

directors, the field operations manager, other QA officers, prime contractors, and subcontractors. This approval information is ty picall y the first page of the QAPP. D epending on the organization’s adm inistrative policy , QAPP approval could also be in a separate memorandum
.QAPP Handbook 17 December 2012 Agency Review Draft The signature dates indicate the earliest date when the environmental data operations portion of the project can start ( i.e., its effective date). The QA Manager will also determine if digital (electronic) signatures are acceptable for the approval of the QAPP. In addition to the title, version number, and approval signatures, it is important to include a revision history . Each time the QAPP is revi sed, as approved b y the QA Manager, the version number should be updated and the revision history should be amended to include a brief summary of the change and date. 2.2.2 Document Format and Table of Contents The QAPP should be organized such that it me ets the project ’ s needs, can be reviewed efficientl y, and meets the document control requirements of the QMS under which it is developed. A document control format, such as the example shown in Figure 4 , or footer created for each page to show revision status may be used to support QAPP development.

Figure 4 . D ocument control format example The Table of Contents will generall y list QAPP elements, as well as any tables, figures, reference s ections , and appendices necessary to the project. If the QAPP author prefers to organize the plan differentl y than how the elements are organized in this Handbook or in the UFP
- QAPP worksheets (EPA 2005b) , a table may be inserted here to cross
- reference where the information for each element may be found to simplify review. S tandard operating procedures (SOPs) may be included as appendices to the QAPP. Depending on the type of project, sampling methods, anal ytical research protocols, or data management procedures may be attached. If SOPs or other data gathering, data analy sis, or evaluation protocols are not documented in, or attached to the QAPP, they

must be available to the project team and QAPP reviewers through some other means. In the case where proprietary s tandards (for example, those of A NSI or IS O ) are used, reference to the location on the relevant web
- site may be sufficient. 2.2.3 Distribution List The distribution list identifies all individuals who should get a cop y of the approved QAPP, either in hard cop y or electronic format, as well as any subsequent revisions. Key personnel responsible for project implementation and/or funding, and who should have the currentl y approved QAPP, should be listed with their project titles or positions, organization nam es, email addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will need access to it, the distribution list also serves as an eas y reference of who needs to be alerted and provided with a revised version of th e QAPP in the case that modifications are necessary . Some organizations choose to provide the distribution list on the title or approval page , o ther s elect to include this list in the project organization section when listing key personnel and their contac t information. Project Name/# __________________ Section # _______________________ Revision # ______________________ Date ___________________________ Page ___________ of ____________QAPP Handbook 18 December 2012 Agency Review Draft 2 .2.4 Project Organization and Schedule It is important that roles and responsibilities are well defined prior to initiating project activities. Those individuals involved with the major aspects of the project are named in this element along with their project responsibilities. For example, the people responsible for maintaining, updating, and overseeing implementation of the QAPP would be named here. The personnel included in this element should include the lead scientists, researchers, mode lers, consultants, and contract and su bcontract QA representatives . If the actual personnel cannot be initially identifie d , then the position description of that person’s function should be given. An organizational chart or table can be very helpful, and should be included if appropriate . It is also helpful to indicate lines of communication among individuals or groups, and this can be shown easily

on an organizational chart or an organizational network diagram. While a single individual may

have more than one responsibility in a project, the project should be organized such that any person having QA responsibilities is independent of those generating and using project information. If this is not pos sible, owing to the size of the organization or for some other reason, an alternative method of ensuring effective QA review should be specified in the QAPP. The level of detail included in the schedule is left to the discretion of the QAPP authors. It may be beneficial to have a very detailed and strongl y s tated sched ule for the project to follow. Th is increases the risk of requiring QAPP revisions if the schedule needs to be changed during the lifetime of the project unless the QAPP explicitly states it w ill not be revised simply due to schedule delay s . Itis useful to include critical points in the project such as expected date of QAPP approval, sub
- section start and end points, dates for sampling, dates by which anal ys es are needed, or dates modeling sub routines need to be completed. When creating the schedule, allowance should be made for potential delays, down
- times, maintenance off
- lines, and general inefficiencies inherent in any project. If the project includes regulatory or court - mandated deadlines , these should be highlighted to ensure their importance is noted. For projects in which milestones or deadlines are not well defined , a more generalized work schedule can be formulated. 2 .2.5 Project Background, Overview , and Intended Use of Data This overview should give the reader an understanding of the problem to be solved, along with any pertinent background information for the project
. Itdescribes why the project will be done and what goals the project intends to accomplish
. The general project goals stated here form the foundation for the entire study
. Equall y important, the development and documentation of this element will help ensure that all project team members clearl y understand and agree on the underlying purpose of the project, increasin g the likelihood that the project design will address and accomplish that purpose. In addition to the general overview , include information on the background and use of previously approved QAPPs relevant to this project. In addition to information about the project, it is good practice to include an outline of information that is currentl y not sufficiently developed. Clearly state who needs the information and what the intended use of this information will be
. Problems that are more complex will lead to mo re extensive information in this section
. The reader should be able to understand the importance or context of the project
. The general project goals stated in this section will be refined in Section 2 .2.6.QAPP Handbook 19 December 2012 Agency Review Draft 2 .2.6 Data

/Project Quality Objectives and Measure ment Performance Criteria EPA requires the use of sy stematic planning for all projects that involve the collecti on or use of environmental data , and the QAPP documents the outcomes of this process (see also Annex C of the two Standards) . EPA encourages th e use of the Data Quality Objectives ( DQO s ) , which may also be called project quality objectives , depending on the organization’s preference
. The DQOs are established from total study variability

as the overall qualitative and quantitative goals of the project. If the DQO process (EPA 2006a) is implemented, the QAPP houses the outcomes of that process. For example, the project background, the number and type of samples needed, and the designs for sample collection and anal ytical measurement are all included in the QAPP. Under any s ys tematic planning process for a project that includes environmental data collection it is desirable to define tolerable levels of uncertainty for components of the total study variability
. Total study

variability

is due to natural (field) variability and measurement (laboratory) variability. The effects of field variability may be minimized by

selecting an appropriate sampling design and number of samples collected (see Appendix B for further discussion). Measurement variability can be minimized by

careful selection of specific measurement techniques (see Appendix C for further discussion)
. DQOs are supported b y D ata Q uality Indicators (DQIs) , which are supported by M easurement Quality Objectives (MQOs)
. These DQIs are linked to field and measurement variability (see Appendix B for further discussion). MQOs are sometimes known as Measurement P erformance C riteria (MPCs) depending on the protocols of the organization
. See Appendix B for definitions and details o f DQI s and MQOs
. Itis essential that the definition of what is meant by MQOs or DQOs is made clear early in this section. Some organizations use the same acronym s for different operations. For example, some organizations use DQOs in place of DQI performance measures, or use MPC s for these measures. During the DQO or other sy stematic planning process, DQIs are considered, and specific MQOs are set to ensure that data are of appropriate quality for their intended use
. These MQOs are the criteria against which the data are measured in the “Do” step of the “Plan
- Do - Check
- Act” cycle
. MQOs/MPCs are the cornerstones of the technical sections of the QAPP. The traditional DQIs considered in project planning include:
 precision;  bias;  representativeness;  comparability ;  completeness; and  sensitivity
. While historically

there has been considerable attention directed to bias, precision, and sensitivity, it is really representativeness that is probably the single most important indicator of data quality
. Representativeness is a qualitative measure of the degree to which data accuratel y and precisely

represent a characteristic of a population
.QAPP Handbook 20 December 2012 Agency Review Draft A poor sampling design wi th very high quality

analy tical laboratory

anal ys es might give data that meet MQOs for precision, bias, sensitivity, comparability , and completeness while not providing information that is actually representative of the characteristics of the population of interest
. Throughout this guidance, the concept of representativeness (as well as the other DQIs) will be used to motivate the need for QA and QC elements. If the data collected fail to be representative of what was intended to be sampled, no amount of st atistical manipulation can make the data set valid
. Great attention should be given to ensure that what is sampled or collected trul y represents the project population of interest. Itis axiomatic that a sufficient number of samples should be taken in orde r to achieve sense of representativeness. For each DQI there should be an indication of how rigorous the measurement needs to be. Without making this clear, data may be inadequate for final use, or conversely , more expensive to collect or produce than nec essary because a higher standard of quality is targeted than is required for the project purposes. The QAPP should indicate how rigorous the collected data or information should be to answer the question, resolve the problem, or support the decision to the level desired and needed for th e project
. For projects that include environmental data collection or production, a cceptance or performance criteria information should be determined during the project planning process and documented in the QAPP
. For examp le, in an enforcement decision, this discussion c ould focus on regulatory or action levels and the quality of the data required to make and enforce decisions in relation to the project action limits
. Some organizations, especially those using the UFP - QAPP , distinguish between screening data (data of known quality suitable for use for interim decisions) and definitive data (data of known quality

suitable for use for final decisions)
. Whether for screening data, definitive data, interim data, or count data, the same tenet holds true: it is important to understand the intended use of the data and to set and meet measurement performance criteria that ensure the quality of that data are sufficient to achieve the decision objectives. It is important to note that the selection of DQOs and MQOs must be grounded in reality. An arbitrary selection of these numerical values, or default to instrument performance characteristics , will rarely lead to the generation of data quality sufficient to meet the needs of the projec t . The MQOs must be clearl y linked to the DQOs, with these in turn linked to the project’s overall objectives. In addition to planning for the ty pe of data needed to address a particular project, it is important to also consider the way

those data will be analyzed. That is, what statistical or other evaluation methods will be applied to the data in order to reach a conclusion for the project? The QAPP should address all aspects of project planning from the problem statement, through data and information ne eds, through quality criteria to be applied, through laboratory anal ys es of samples, through statistical analy ses of the data, through the final project decisions and reports, to disposition and maintenance of the data and records. Establishing DQOs and M QOs for exploratory projects (for example, research studies) where little or no information on the quantitative magnitude of what is expected is difficult. The use of information from similar or related studies is useful.QAPP Handbook 21 December 2012 Agency Review Draft For example, consider an experiment designed to investigate the neutralizing properties of a chemical on a known contaminant mixture. For this experiment, it may be sufficient to use qualified statements such as “the precision (MQO) for measuring reduction of acid is expected to be of the order +/
- 10% based on similar experiments as documented in XXXX”. A discussion documented in the QAPP of the consequences (or results) of achieving different realized DQOs or MQOs is advisable. In some circumstances, DQOs may be stated in qualitative term s having definitive reference points. For example, “The project will produce data that will qualify to receive the ‘A’ rating with respect to the rating sy stem described in Section 4.4.2 of the Procedures for Preparing Emission Factor Documents (EPA
- 454/R
- 95 - 015)” (EPA 1997a). Although it is stated in qualitative terms, this DQO is measurable using specific acceptance criteria that are presented in the cited document. Simplistic descriptors such as “very good” or “acceptable for this project” are to be dis couraged unless these are specificall y defined in the QAPP. 2 .2.7 Special Training Requirements and Certification Special training or certifications are sometimes necessary for project personnel and laboratories associated with projects
. This may include s uch things as having project personnel complete specialized hazardous material handling training , being skilled in the collection of samples for trace metal anal ys is, being trained in global positioning technology , or being trained in specific sampling techniques
. Similarly , project personnel may require special security

clearances such as clearance to enter a military site
. Laboratory certification for the anal ys is of certain types of samples may

also be necessary
. The QAPP should describe how the proje ct will ensure that competent personnel are available to perform the work
. A method for ensuring competency helps ensure project specific requirements are met. For example, the ORD Policies and Procedures Manual, Section 13.4, Minimum QA/QC Practices for O RD Laboratories (EPA 2001b) , states “Prior to performing sample anal ys is with a new method for which proficiency has not been previously

demonstrated, the anal ys t must demonstrate proficiency with the method by completing the following: (1) perform valid i nitial calibrations, (2) perform method detection limit determination, (3) demonstrate that they can meet all minimum QA/QC acceptance criteria as presented in the method document, e.g. the SOP, and (4) if available, satisfactoril y anal yz e a performance ev aluation sample or a second source standard”. The QAPP should also s pecify how this information will be documented, where the records will be kept, and indicate who is responsible for ensuring that the se special training and certificate needs are met and d ocumented
. 2 .2.8 Documentation and Records Requirements The QAPP should d escribe the process and responsibilities for ensuring that project personnel will receive the most recently approved project documents such as the QA PP , S OP s, and other documents used throughout the project operation
. The QAPP authors may choose to t ell how these documents will be updated and how information regarding updates will be communicated
. The QAPP should i dentify the information to be included in the project data files (o r project record) and its format
. This might include the following items :
 sampling collection and handling records such as o field notebooks or operational records ,QAPP Handbook 22 December 2012 Agency Review Draft o Global Positioning System (GPS) data , o chain
- of

- custody forms , and o sample receipt records, inc luding sample tags and shipping bills ;  anal ytical records such as o anal ytical log books, o test method raw data and QC sample records, o Standard Reference Material and/or proficiency test sample data , and o instrument, equipment, an d model calibration information; and  data assessment records such as o validation of software used, o input and output files as results of code , o statistical methodologies, and o data
- base test procedures. In some cases, access and the final disposition of records may be regulated o r required b y policy and this status should be included in the QAPP. The QAPP should c learl y i ndicate where all project documents , report s, records ( including digital records ) will be stored and for how long
. **2 .3**  **DATA ACQUISITION (D**  **O )**  This is the heart of a QAPP as it documents the requirements for sample identification (where the samples will be taken), sample collection (how to preserve the integrity of each sample), and sample handling (the management of the samples). This section of fers advice on the Standards Clause 2.1, Scope, Clause 7.7, Systematic Planning, and Annex B3.4, Data Acquisition. The elements in this section address all aspects of data production and collection to ensure that appropriate methods for sampling, measurem ent and anal ys is, data production , data handling, and QC activities are employed and documented
. The se elements describe the actual methods or methodology

to be used for the collection, handling, anal ys e s of samples, and the management such as compiling and handling of the data. Where applicable, the elements shall be addressed in the QAPP and include:
 sampling collection design, experimental design, and sampling tasks;  sampling procedures and requirements;  sample handling and custody requirements;  anal y tical methods requirements;  quality

control requirements;  instrument/equipment testing, calibration and maintenance requirements, supplies and consumables; and  data management requirements. The QAPP should provide detailed information on the methods to be used
. It is important to note that i f the designated methods are well documented and are readily available to all project participants, citations are adequate
.QAPP Handbook 23 December 2012 Agency Review Draft When procedures or methods are not commonl y used or readily available , detailed copies o f the methods and SOPs should accompany the QAPP either in the text or as attachments. 2 .3.1 Data Collection Procedure , Experimental Design , and Sampling Tasks This element describes the project’s data collection or research experimental design
. Key s to this element are the assumptions made and how the data will be obtained
. This element explains the “how and wh y” of the project’s information collection design to ensure that the appropriate data are collected for this project
. Input for this element will come from the project’s s ys tematic planning (e.g., the DQO p rocess ) and also see Annex C of the two Standards
. For data collection th ere are two classes of sampling designs to consider: probability
- based and judgmental , and they

have very

different properties. The former are sometimes called statistical designs , and the latter sometimes called directed sampling designs
. Strong statistical conclusions are available with probability
- based designs but not with judgmental designs
. Use of professional expertise and/or historical knowledge about the site can improve development of statistical or judgmental sampling designs
. Key questions to be considered include :  i s this project to be comparable with previous sampling or analy tical efforts, or with a health
- base d or regulation standard?  c an samples or measurements be taken according to a probability
- based design?  i s the objective of the sample to estimate an average or to find a hot spot?  i s there a reference or background population that can be used as a compari son to the target population?  w ill sampling units be chosen in advance , in the field based on visual evidence , or using other technical evi d ence ?  i s there a network of sampling sites that will be visited periodicall y or where sampling will be performed continuously?  d o all the samples need to be taken simultaneously?  i s the target population approximately homogeneous or is it heterogeneous in nature needing stratification or division into approximately homogeneous areas?  c an samples be composited? and  are there any

potentially

important sources of variability which affect the sampling such as tidal cycles, seasonal differences, and rain and wind patterns ? The answers to these questions should have been considered during the planning process and help to determine allocation of resources for obtaining samples for anal ys is
. For example, when the project involves a phys ical site, this element of the QAPP may involve:
 defining the size of the area, shape, volume, or time that is to be represented by a sample (called the scale of representativeness) as part of the justification for how the sampling sites and durations will be selected;QAPP Handbook 24 December 2012 Agency Review Draft  specifying the type and number of samples to be collected, and their locations; and  providing detail ed schedule s for sampling and analy tical activities, test runs, and reviews. If, instead, the project is to develop a new anal ytical measurement method, this step could involv e:  defining the procedure to be used for the testing ;  specifying the levels at which testing will occur ;  setting a timescale for sequential instrument readings; and  clearl y stating any extraneous variables (e.g., temperature) that will be controlled. The QAPP should describe the experimental data production or data collection design for the project, including as appropriate:
 the sampling design (e.g., sy stematic grid if spatial, time
- interval if temporal) ;  the rationale for the design;  the ty pes and numbers of samples required ;  restrictions due to resource constraints;  need to collect background samples;  the samp ling locations and frequencies;  rationale for ensuring representativeness of data; and  mea surement paramet ers of interest. Advice on selecting the appropriate sampling design may be found in Chapter 2 of the Guidance for Choosing a Sampling Design for Environmental Data Collection , EPA QA/G
- 5S (EPA 2002 b ). Advice on experimental data collection is available from the standard s tatistical literature under the title “Design of Experiments” or “Experimental Design”. 2 .3.2 Sampling Procedures and Requirements Standardized sampling procedures provide consistency between samplers; facilitate collection of accurate, precise, and repres entative samples; and help to ensure data comparability and usability
. These may

be in the form of SOPs established by the organization or may

be unique to the investigation in hand. The QAPP should provide a list or table of all field sampling procedures and brief descriptions of sampling procedures. In many instances reference to where the SOPs are located should be sufficient but in some cases may be attached to the QAPP to allow for review and approval
. Any SOPs that are modified to meet project
- specifi c needs should be attached to the QAPP. Sampling procedures should include SOPs, when available, for sampling each matrix and each anal ytical parameter for each type of equipment and technique
. The SOPs provide detail on the appropriate number, size, and ty pe of sample containers to be used for collection of each field sample and field QC sample and the proper temperature, light, and chemical preservation procedure for those samples.QAPP Handbook 25 December 2012 Agency Review Draft Also included should be information on the techniques to be used for spe cific locations due to difficulties i n sample collection. These can include restrictions on phy sical sample collection (e.g., inability

to collect the sample where planned) , availability of trained personnel (e.g., lack of a certified technician) , or seaso nal influences on the obtaining of a phy sical sample (e.g., adverse winter conditions)
. The QAPP should describe how samples will be collected
. The selected sample collection procedures should be appropriate to ensure that project personnel collect representative samples in a consistent manner for all required sample matrices and locations ; that contamination is not introduced during collection ; and that sample volumes are properl y preserved in order to meet project objectives. The QAPP should include a des cription of preservation procedures (temperature, light, or chemical) that maintain sample integrity in the field, prior to and during shipment to, and immediately

upon receipt by

the off
- site or mobile on
- site laboratory
. The QAPP should document requirem ents for sample volumes, container ty pes, number of containers, and preservation procedures for each anal ytical group, matrix, and concentration level. The QAPP should provide details on the procedures for both the initial cleaning of sampling equipment an d also subsequent decontamination procedures that will be followed during the sampling event
. These procedures will help ensure that collected samples are representative of the sampling location b y verifying that sampling equipment is clean and free of target anal yt es , C hemicals of Concern, or interferences
. Cleaning and decontamination procedures should cover all equipment that contacts a sample
. If the sampling equipment is disposable (“one use only”), procedures for cleaning and decontamination are not necessary ; however, the QAPP should state that disposable equipment will be used and where that equipment will be deposited. Development of a standardized table customized to the needs of the organization is encouraged as it leads rapid review and ease in tracking adherence to sample anal ys es requirements. 2 .3.3 Sample Handling , Custody Procedures , and Documentation The QAPP should include all sample collection documentation and sample handling, tracking, and custody procedures needed to ensure that sample integrity and custody are maintained
. Without sample integrity, and without clarity regarding the chain
- of

- custody for all samples, it may be impossible to assert that the data are of adequate quality to meet project objectives
. The procedures should addre ss sample collection, packaging, handling, and shipping, as well as records, receipt of laboratory samples, archiving, and disposal
. These may be included by reference if an approved SOP is maintained by the organization. It is important to note that the d etail needed for handling, chain
- of
- custody, and documentation should be commensurate with the goals or objectives of the project (application of the Graded Approach). There are many sample collection procedures that do not need extensive procedures (for example, use of a sample inventory , or use of an EPA number to preserve some degree of confidentiality) . Reference to the organization’s Quality Management Plan and the organization’s QA Manager should clarify what is needed to meet the project’s objectiv es.QAPP Handbook 26 December 2012 Agency Review Draft Sample Handling and T

racking Proper sample tracking systems support the chain
- of
- custody procedures, which help to ensure sample authenticity

and data defensibility
. The QAPP should document the procedures that will be followed to identify

and track samples that are collected in the field, analy zed on
- site, and delivered or shipped to an off
- site laboratory

for analy sis, as well as samples transferred throughout the laboratory
. If samples are shipped to an off
- site laboratory , then the laboratory ’s samp le handling and tracking s ys tem should also be described. The sample handling and tracking procedures should include the following descriptions :
 the sample numbering s yst em for field sample collection with an example ( i f applicable, the numbering s ys tem should follow specific programmatic requirements that apply to the project
. A s ys tematic approach for numbering samples should be used so that each sampling location, matrix ty pe, sample depth or height, and date and time of collection can be uniquel y identified and cross
- referenced to the programmatic sample number) ;  the sample contai ner identification information;  the latitude, longitude, and elevation (altitude) for the sample collection location;  the laboratory

sample tracking procedures ( i f laboratory identification numbers will be used to track samples internall y, the laboratory procedure must describe how these laboratory

identification numbers will be cross
- referenced with the samp le number assigned in the field) ; and  the sample storage procedures us ed b y the off
- site or mobile on
- site laboratory
. To demonstrate the project’s sample handling process, the QAPP should include a table that shows the flow of samples from the time of collection to laboratory delivery to final sample disposal
. The table sho uld identify each component of the project
- specific sample handling sy stem; indicate the personnel (and their organizational affiliations) who are primaril y responsible for ensuring proper sample handling, custody , storage, and disposal; and specify the length of time that samples, digestates and extracts, and biological collections will be retained by the laboratory

prior to disposal. The QAPP should also describe how samples will be delivered or shipped to the laboratory . The description should include th e name of the carrier service, if applicable, and define how samples will be batched or grouped when sent to the laboratory . The QAPP should include provisions for packaging, marking and labeling, and shipping samples in compliance with the Department of Transportation regulations for shipping hazardous and nonhazardous materials (see also 49 CFR 171)
. Air carriers that transport hazardous materials require compliance with the current edition of the International Air Transport Association Dangerous Goods Re gulations. Shipment papers, including bills of lading and airbills, should be retained by the laboratory with chain
- of
- custody records. Examples of all sample shipment forms to be used should be attached as an appendix to the QAPP
.QAPP Handbook 27 December 2012 Agency Review Draft Sample Custody Procedures A sample is in “custody” if it is in the actual phy sical possession of authorized personnel or in a secured area that is restricted to authorized personnel
. For some projects, an evidentiary paper trail documenting sample custody is required in order to meet project quality objectives
. Since it is often difficult to predict what samples or projects will require proof of custody after the fact, all data collection events should employ documented chain
- of

- custody procedures to ensure data authenti city

and defensibility
. The evidentiary trail from sample collection through data production and archiving is maintained using sample custody procedures and documented by complete chain
- of

- custody records
. Chain
- of

- custody procedures ensure accountability for the location and integrity

of the sample at all times
. The ASTM document D4840
- 99 Standard Guide for Sampling Chain
- of
- Custody Procedures contains useful information regarding chain
of
- custody procedures (ASTM 1999 )
. The QAPP should describe (or have attached) the procedures that will be used to maintain sample custody

and integrity

and to document the implementation of chain
- of
- custody procedures
. Sample custody procedures should include the field sampling team’s procedures for maintaining and documenting sample custody from the time samples are collected in the field through packaging, shipment, and delivery to the laboratory
. Field sampling documents that describe chain
- of

- custody procedures, including SOPs, should be attached to or referenced in the QAPP . The laboratory ’s procedures for maintaining and documenting sample custody from the time the samples are received at the laboratory through archiving and disposal should also be attached to or referenced in the QAPP
. The use of identification or trac king software such as FORMS II Lite TM , should be documented, if applicable
. Examples of all chain
- of
- custody document s that will be used during the project should be provided in the QAPP, including chain
of
- custody forms, sample identification, custody seals, laboratory sample receipt forms, and laboratory

sample transfer forms. Sample Documentation The QAPP should describe sample documentation procedures that will be followed for the project. Proper documentation of field sampling activities helps ensure s ample authenticity (i.e., the sample identity

is correct) and data integrity. The QAPP should also indicate what information should be marked on the sample collection container label (e.g. time, date, sample number), the anal ys es to be performed, preservatives used, and name (with signature ) of the field technician completing the sample collection. Documentation for sample collection includes, descriptors of sample type (duplicate, QC check, etc.) , field logbooks, field data collection forms, field calibration forms, chain of custody , and sample container identification. The QAPP should specify the required sample identification information and include an example. An electronic sy stem, such as a bar code or FORMS II Lite TM (which retrieves information store d elsewhere), may be used. The QAPP should also describe how the information on the label will be preserved (e.g., b y covering the label with clear tape to minimize water damage during transit).QAPP Handbook 28 December 2012 Agency Review Draft 2 .3.4 Analy

tical Methods Requirements and Task Description The QAPP should i dentify the anal ytical methods , sensitivity, precision, bias and acceptance limits for QC results, equipment required, sub
- sampling or extraction methods, laboratory decontamination procedures and materials (such as in the case of hazardou s or radioactive samples), waste disposal requirements (if any), and any specific performance requirements for the method
. Where appropriate, anal ytical methods may be identified using their number, date, and regulatory

citation
. If data from laboratory an aly ses do not form part of the overall project description, then a simple outline of the data production method is necessary . For example, to determine if there have been observable ecological deficits at a study site, plant cover and a diversity study sho uld be included in the project summary . The QAPP should then describe the number and size of the study sites, the procedure for counting the species (e.g., number of unique species or number of specimens of each species), and the procedure for locating the sites. The QAPP should specify any m ethod performance standards
. If a method allows the user to select from various options, then the rationale for selection should be given
. For non - standard method applications such as for unusual sample matrices (e.g., explosives testing of sea cucumber tissue) , appropriate method performance study information is needed to confirm the performance of the method for the particular matrix
. If previous performance studies are not available, they should be developed during th e project and included as part of the project results. The organizations or laboratories that will provide the anal ytical services (all on
- site and off
- site laboratory

anal ytical work, including all prime laboratories, subcontractor laboratories, an d back up laboratories) should be identified together with the measures taken to ensure laboratory proficiency
. Itis recommended that a table of analy tical services be included in the QAPP, containing the following information at a minimum :
 matrix;  anal ytical group or class of compounds ;  concentration level;  sample location or identity number;  anal ytical SOP to the analy sis requested;  preservation and holding time;  number and type of sample containers ;  required data package turnaround time;  the number of QA and QC samples needed;  laboratory

/organization (name, address, contact person, and phone number); and  the arrangements made with a backup laboratory/organization. If these method requirements are already described in the laboratory’s set of SOPs, these should be attached or referenced as appropriate. In addition, the format of the data package (electronic, hard
- cop y , etc.) should be identified. These requirements should be commensurate with those requirements contained in the overall agreement between the organization and laboratory ,QAPP Handbook 29 December 2012 Agency Review Draft including the timeframe for delivery of data and information . It is possible th e conditions of the agreement will affect the MQOs established earlier and resolution of the differences will be needed (for example, the detection li mit MQO set at 0.1ppb, but the agreement reporting limit set at 0.5ppb)
. The measures taken to ensure the laboratory will meet the proposed data quality needs of the project should be discussed. When necessary , as determined b y application of the graded ap proach, the review of SOPs and the results from performance evaluations using similar contaminants of concern in matrices similar to those expected may be needed. It is important to address what to do when a failure in the anal ytical sy stem occurs, who is responsible for corrective action, and how the effectiveness of the corrective action shall be determined and documented. 2 .3.5 Quality

Control Requirements Quality Control (QC) has two important aspects: preventive and corrective. The preventive aspect is designed to prevent quality problem s before they

are observed. This is achieved through the use of proactive procedures that minimize variability and bias at each stage of measurement. Examples of proactive QC include the use of check sheets, control c harts, and standardized data recording forms to promote and document consistency in performance of specific activities. The c orrective aspects of QC can sometimes alleviate situations where there is the potential for an error in interpretation
. Some of the retrospective activities of data verification and validation may

be considered examples of corrective action , whereas the use of statistical (Shewhart) control charts is an example of proactive corrective activities. The QAPP should identify QC activities needed for each sampling, anal ys is, or measurement technique
. For each QC activity , the associated method or procedure, acceptance criteria, and corrective action should be listed
. As standard methods are often vague or incomplete in specifying QC require ments, simply relying on the cited method to provide this information is usually insufficient
. QC activities for the field and the laboratory include, but are not limited to, the use of blanks, duplicates, matrix spikes, laboratory control samples, surrogates, or second column confirmation
. A list of commonly used QC samples is included in Appendix C
. In environmental studies, measurement is the process of obtaining a quantitative value describing a chemical or phys ical property of an individual sampling unit or specimen collected from this unit
. Measurement may

involve direct field measurements using survey instruments or collection and handling of ph ys ical samples followed by analy sis in a fixed or mobile laboratory
. Choices that the technical team makes regarding sample acquisition, sample handling, preparation, and anal ys is can influence the quality attributes of the resulting data
. Sources of measurement variability

include:
 within
- unit, small
- scale variability (influenced b y the nature and distributio n of the characteristic of interest within the sampling unit and the media assayed);  phy sical sample acquisition protocol (sample collection tools and procedures such as compositing);  sample handling and transport;QAPP Handbook 30 December 2012 Agency Review Draft  variability

associated with the measure ment assay (influenced b y the capabilities and management of the anal ytical resources);  inaccurate determination of strata boundaries;  inadvertent bias b y the sample selector (identification uncertainty);  homogenization and sub
- sampling procedures;  sample preparation , extraction procedures , and sub
- sampling extract for analy sis; and  anal ytical determination
. Each MQO should have at least one QC procedure designed to prevent unsatisfactory results for as many sources of variability

as possible. An example would be the proper adherence to documented SOPs (a preventative measure) combined with use of Shewhart QC charts (statistical process control) to monitor analy tical results (a corrective measure). The QAPP should d escribe , reference , or attach the procedures to be used to calculate applicable statistics such as precision and bias
. Copies of the formulae are acceptable as long as the accompanyi ng narrative or explanation specifies clearl y how the calculations will address potentially

difficult s ituations such as missing data values, “less than” or “greater than” values, “below detection”, “not detected” and other common data qualifiers. One use of routine QC results is assessment of the laboratory's internal operations; for example, the routine anal ys is of calibration blanks to test for inorganic contamination or a problem in sy stem calibration. In this case, a DQI could be just the actual blank result, expressed in appropriate units, with an MQO equal to the laboratory’s internal acceptance crite rion. A poor result from a routine calibration blank should result in an immediate corrective action by the laboratory . This action may be limited to simply

rerunning a calibration blank to demonstrate that the first result was an aberration. While it is important to verify that the laboratory

is monitoring and controlling the quality of their internal operations, results from samples like a calibration blank are less important in project planning from the DQO perspective than other potential sources of error that may not be controlled. The same can be said for overall anal ytical calibrations. A properl y calibrated analytical system is always required and expected for anal ys is of project samples. 2 .3.6 Instrument/Equipment Testing, Calibration and Maintenan ce Requirements , QA/QC for Supplies and Consumables It is necessary

to recognize the growing importance of field instrumentation and use, and so the same care and attention given laboratory equipment should be given to field equipment. If the field equipm ent is intended to give less precise readings, lesser QA/QC requirements may be needed as documented in the organization’s QMP. Many of these requirements are part of a laboratory ’s quality s ys tem and should be documented in the laboratory ’s quality manual (however named). It may be sufficient to simply

reference this quality manual (including revision number and date) or reference the contract arrangement between organization and laboratory . If the project has specific requirements different from the laboratory ’s quality

manual, specificity in activities, actions, anal ys es, and responsibilities for their successful implementation may be necessary . The depth of specificity should be linked toQAPP Handbook 31 December 2012 Agency Review Draft the project’s objective b y application of the graded approach or c onsultation with the organization’s Quality

Manager. Instrument

/Equipment Testing and Inspection The QAPP should d escribe how inspections and acceptance testing of instruments, equipment, and their components affecting quality will be performed and documented to assure their intended use as specified
. The QAPP should i dentify and discuss the procedure by which final acceptance will be performed by independent personnel (e.g., personnel other than those performing the work) and/or by the EPA project m anager . The QAPP should d escribe how non conformances or deficiencies are to be resolved, when re
- inspection will be performed, and how the effectiveness of the corrective action will be determined and documented. Instrument Calibration and Maintenance The QAPP should i dentify all tools, gauges, i nstruments, and other sampling, measurement and test equipment used for data collection or production activities affecting quality that should be controlled and calibrated at specified periods, to maintain performa nce within specified limits
. The QAPP should include detailed d escriptions or reference s to SOPs describing how calibration will be conducte d , the frequency of calibration, the procedures for anal yz ing calibration data, the acceptance criteria for calibration limits (with references), what certified equipment is needed, how valid traceability to established performance standards will be demonstrated, and how long calibration records will be maintained. If no such establis hed standards exist, the basis for the calibration should be documented
. The principal reason for performing calibration is to reduce bias to a minimum. The decision as to the ty pe of standard the instrument should be calibrated against depends on the overall accuracy expected from the instrument, the project objectives (DQOs or MQOs as appropriate), and the resources allocated for this part of the project’s development. The DQOs/MQOs of the project may

be badly compromised should an inadequate calibration be performed. Ineffective calibration will often give the impression of precision which can be substantially overwhelmed by an increase in bias. See Appendix C for a further discussion on bias. The QAPP should d escribe or reference the process for how periodic preventive and corrective maintenance of measurement and test equipment or other s ys tems and their components affecting quality shall be performed to ensure availability and satisfactory

performance of the s ys tems . The QAPP should i dentify the equipm ent or s ys tems requiring periodic maintenance
. The QAPP should also discuss how the availability of critical spare parts, identified in the operating guidance and/or design specifications of the s ys tems, will be assured and maintained. Supplies and Consuma bles Not all projects will need supplies and consumables considered “critical” to the project. For those that do, this element documents the sy stem used to ensure appropriate materials are obtained in a timely

fashion. The QAPP should describe how and b y whom supplies and consumables (e.g., standard materials and solutions, sample glassware/containers , calibration gases, reagents, hoses, deionized water, potable water, and electronic data storage media) shall be inspected and accepted for use in theQAPP Handbook 32 December 2012 Agency Review Draft project . The QAPP should state acceptance criteria for such supplies and consumables and have criteria for disposable sampling equipment and supplies
. Where applicable, information enabling the needed materials to be located and obtained should be listed in the Q APP, including:
 supply

source or vendor;  procedures for identifying, tracking, storing, and retrieving these materials;  identification of those responsible for maintaining supplies; and  any acceptance criteria needed (for example, certificate of testing). 2 .3.7 Data Management Requirements The QAPP should d escribe the project data management process, tracing the path of the data from their production to their final use or storage (e.g., the fie ld, the office, or the laboratory), which is very

important for ensuring data integrity
. The QAPP should d escribe or reference the standard record
- keeping procedures, document control sy stem, and the approach used for data storage and retrieval of electronic media
. The QAPP should discuss the control mechanism for dete cting and correcting errors and for preventing loss of data during data reduction, data reporting, and data entry

to forms, reports, and databases
. E xamples of any forms or checklists to be used should be provided
. The QAPP should i dentify and describe all data handling equipment and procedures to process, compile, and anal yz e the data , including any required computer hardware and software , and address any specific performance requirements for the hardware/software configuration used
. A d escription of the p rocedures that will be followed to demonstrate acceptability of the required hardware/sof

tware configuration should be included (for example, after upgrading software, the data should be checked to ensure the migration has occurred successfull y)
. Finally , the QAPP should discuss, as necessary , the archiving, data migration, and software update changes checks that will be used to ensure no data loss. This i nclude s backup procedures for any data stored electronically , and p rotocols for access to, retrieval from, and photocopying of, information archives. In some cases retention, access to, and final disposition of some records may be regulated and require field and laboratory QC results to be captured in electronic data management s ys tems. In those cases, th is element should address and comply with all relevant regulations. In other cases, requirements for this element may be stated in the terms and conditions of external agreements and the QAPP should identify any requir ements set by

statute or policy
. Furth er assistance on aspects of data management may be found in Brilis et. al ., 2004. **2 .4**  **ASSESSMENTS (C**  **HECK )**  The third part of the Plan
-Do
- Check
- Act paradigm considers the technical aspects of QA that apply to ensuring the integrity of the data. Not all of these different assessments are necessary and should be selected by application of the graded approach that selects the level of assessment commensurate with the intended use of the data. The advice given here is drawn from the Standards Clause 7.8, Assessment of Data and Information, and Annex B3.5 Assessments. These QAPP element s ensure that planned project a ctivities are implemented as described in the QAPP and that reports are provided to apprise management of the project status and any QAQAPP Handbook 33 December 2012 Agency Review Draft issues that arise during implementation (see also Section 2.5.5)
. Assessment activities help to ensure that the resultant data quality is adequate for its intended use, and that appropriate responses are in place to address non - conformances and deviations from the QAPP
. Several ty pes of assessments that could be included are:
 technical s ys tems assessment s;  performance audit s of measurement and analy tical sy stems ;  surveillance operations;  audits of data quality;  qualitative and quantitative comparisons to acceptance and other criteria;  interim assessments of data quality ;  evaluation of unconventional measurements; and  evaluation of unconventional monitoring projects
. If deviations from the QAPP are identified through a formal assessment or audit, corrective actions may

be necessary; these are discussed in greater detail in Chapter 3, Section 3.5 of Guidance on Technica l Audits and Related Assessments for Environmental Data Operations , EPA QA/G
- 7 (EPA 2000). Deviations from the QAPP identified through formal or informal review may

also require corrective actions but, using the graded approach, the project manager and QA Manager should confer to ascertain the possible impact on the validity of the data. Their joint decision should be documented in the revised QAPP. In some cases deviations from the QAPP can be identified by project personnel without the benefit of formal, scheduled assessments . This section addresses those situations and describes the process by which the need for corrective action is documented, reported, and implemented and its effectiveness assessed
. 2 .4. 1 Technical Sy stems Assessment s Technical S ys tem s Assessments ( TSA s) can have two forms: field
- based and laboratory
- based
. A field TSA includes an auditor monitoring of sampling activities to ensure that the QAPP and any field sampling and sample handling SOPs are being followed. Additionally , the fi eld TSA should evaluate reporting, handling, tracking, management, and review of data generated in the field (e.g. pH, salinity , conductivity , test kits, and GPS data). A field TSA may

be conducted independently of the laboratory

TSA and may

even use diff

erent personnel (auditors). A field TSA can be performed at the start of field sampling activities as a readiness review. The readiness review is a technical assessment that is planned and performed prior to the initiation of a project to verify that proje ct management has brought the facility and applicable measurement sy stems to a state of readiness to begin the project. A laboratory

TSA is a thorough assessment of laboratory analy tical procedures during which the facility ( or mobile lab oratory), equipment, instrumentation, supplies, personnel, training, anal ytical methods and procedures, laboratory procedures, sample handling and tracking, data reporting, data handling and management, data tracking and control, and chain
- of
- custody procedures are checked for conformance with the QAPP
.QAPP Handbook 34 December 2012 Agency Review Draft There are several way s of conducting a laboratory TSA . The technique of tracing the project samples through a laboratory can be used effectively

to gain an understanding of the overall anal ys is process during a T SA . If required b y the QAPP, samples should arrive with chain
- of
custody papers or appropriate documentation and should be checked for discrepancies needing resolution before further anal ytical work is done
. Ty pical sample handling steps may be digestion, extraction, and splitting or making an aliquot of the extract for various anal ys es . One way to trace an assessment trail is to physically follow the trail of the samples through the facility , looking in the records for their fate at each step of the trail
. An laboratory

TSA can be performed prior to, at the start of, or at any time during field
- sampling activities. A TSA performed during the data collection phase of a project may be particularl y helpful in discovering operational problems earl y that could adversely impact the quality

of the project results and the usability of the data. The decision to conduct a field or laboratory TSA , and the intensity of the TSA, should be linked to the project’s objectives and intended use of the data. It is important that the principle of the graded approach be considered and the extent of the TSA tailored to avoid unnecessary expenditure of resources. 2 .4. 2 Performance Audits of Measurement and Anal yt ical Sy stems This has two components: proficiency evaluation, and split sample anal ys is. Proficiency Evaluation A performance evaluation ( PE ) sample is a quantitative assessment in which anal ytical results are generated b y a measurement s ys tem for a sample that originates outside of a project
. PEs may also b e known as proficiency testing (PT) samples
. Statistical analy sis of PE sample results provides information on routine laboratory performance and the overall accuracy and bias of the anal ytical method
. Id eall y, a PE sample mimics routine field samples in all po ssible aspects, including similarity of container, label, and shipping, such that the anal ys t is unaware that it is not a routine project sample
. When possible, PE samples should be of similar matrices and analytes, in the same approximate concentrations a s expected in the field samples, so that the analy st does not have to treat the PE sample differently than typical project samples (e.g., does not have to dilute the PE sample if project samples don't have to be diluted). Sample matrix similarity is especi all y important for projects involving solids, sediments, biosolids, and tissues. One potential drawback to PE samples is the difficulty in mimicking the complexity of the matrix containing a chemical of concern. If the PE matrix is different from the actual sample matrix, difficulties in interpretation of readings can occur. In the context of a Quality Program , a PE is used to determine if a measurement sy stem’s results are within data quality goals specified in the QAPP
. PE results are often used to estimate the degree of bias in the measurement sy stem . Although a PE sample can identify

a problem quantitatively , it ty picall y cannot be used to directly determine the cause of the problem
. The appropriate choices for the PE samples’ composition should be speci fied in the QAPP
.QAPP Handbook 35 December 2012 Agency Review Draft When needed, the QAPP should address the selection of appropriate PE samples
. Factors to consider include :
 anal yt e selection;  whether P E samples are single
- or double
- blind, native or sy nthetic matrix, spiked or natively contaminated or both;  multiple matrices and concentrations;  total number of P E samples;  acceptance criteria for PE samples; and  anal ytical methods. A plan for selection, implementation, and reportin g of PE samples should be included in the QAPP. Split Sample Analysis One use of split sample analy sis is for comparison studies conducted to assess inter
- laboratory precision and accuracy . The sampler collects one field sample and then phy sicall y split s it into two representative sample aliquots. The samples are then sent to different laboratories for anal ys is. Split samples quantitatively assess the measurement error introduced by the organization’s sample shipment and anal ys is sy stem. Split sample comparability criteria must be generated prior to sample collection and documented in the QAPP. Another use for split sample anal ys is is for projects involving enforcement actions or compliance with a set standard. Split samples are collected and one sample is anal yz ed by the EPA, the other is provided to a party representing the site or facility owner. Here the two samples are independentl y anal yz ed although EPA may not know the results of the other sample (including QC results) until much later in the proj ect’s development. For further information on the use of audits see Guidance on Technical Audits and Related Assessments for Environmental Data Operation, EPA QA/G
- 7 , (EPA 2000). For further information on the use of various QC samples ( including split samples ) , see Appendix C of this guidance
. 2 .4.3 Surveillance of Operations Surveillance is the observation of ongoing work for the purpose of verifying and documenting conformance with specified requirements and/or procedures such as those given in the QAPP and / or SOP . Surveillance is focused on a particular technical activity , rather than on the entire measurement s ys tem . It is ty picall y less formal than other types of assessment s, but it should also include appropriate preparation, conduct, reporting, and follow
- up phases
. As appropriate, surveillance may

be employed as part of a TSA
. The objective of surveillance is to provide confidence through real
- time observations that an activity has been performed in accordance with approved and specified methods and procedures
. It allows for immediate identification of any deficiency

and initiation of action to correct the deficiency and its underlying cause
. When non - conformanc es or deficiencies are identified, project management should be notified promptly so that corrective action may be implemented
.QAPP Handbook 36 December 2012 Agency Review Draft The QAPP should also state under what conditions surveillance of operations will be performed, ( i.e., what are the triggering events that would require a surveillance assessment)
. The organization assessed should provide a written response that discusses the action taken to correct any observed deficiencies and to prevent similar deficiencies in the future
. The QAPP may allow for immediate notification of the status and performance of the project to management if authorized by

the scope of the assessment . It also can include follow
- up to verify that corrective action was implemented
. In some cases, activities observed during a sur veillance assessment may merit an immediate stop - work order
. The QAPP should discuss the circumstances that would cause a need for such an action and the procedure for issuing an official stop - work order ( Internal Standard, Clause A3.12 (EPA 2013a)
. 2 .4. 4 Audits of Data Quality An audit of data quality is an examination of data after they have been collected and verified by project personnel
. It is conducted to determine how well the measurement sy stem performed with respect to the performance goals spec ified in the QAPP and whether the data were accumulated, transferred, reduced, calculated, summarized, and reported correctl y . It documents and evaluates the methods by which decisions were made during treatment of the data
. The QAPP should specify

the qu e stions to be answered in an audit of data quality such as :
 i s there sufficient documentation of all procedures used in the data collection effort to allow for repetition of the effort by a person or team with technical qualifications similar to those of the original data collector?  c an the data be replicated by the original data collector?  i s there sufficient documentation to verify that the data have been collected and reported according to these procedures?  i s enough information provided to allow a potential user to determine the quality and limitations of the data and whether the intended use of the data is appropriate? and  are the data of sufficient quality with respect to DQI goals and other performance criteria for their intended use? An audit of d ata quality entail s tracing data through their processing steps and duplicating intermediate calculations
. A representative set of the data is traced in detail from raw data and instrument readouts through data transcription or transference through data manipulation (either manually or electronically

by commercial or cu stomized software) through data reduction to summary data, data calculations, and final reported data
. The focus is on identifying a clear, logical connection between the steps
. Particular attention is paid to the use of QC data in evaluating and reporting the data set
. For a large project, a statistical approach may be necessary to determine a representative number of datasets to be examined
. Often, however, the number of sets is limited by

the budget of the a

udit
. An audit of data quality may occur at dif

ferent stages of the project
. For example, it may be performed after field anal ys is but prior to off
- site laboratory anal ys es . The products of the data quality assessment are a report detailing the results of custody tracing, a study of data transfer and i ntermediate calculations, a review of QA and QC data, aQAPP Handbook 37 December 2012 Agency Review Draft study of project incidents that resulted in lost data, and a review of study statistics
. The a

udit report typicall y ends with conclusions about the quality of the data from the project and their fitness for their intended use. The scope and schedule (either b y date or triggering event) for any audit of data quality should be set in the QAPP. 2 .4. 5 Qual itative and Quantitative Comparisons to Acceptance Criteria The data now need to be assessed against the acceptance criteria already established. For each DQI or parameter for which acceptance criteria has been established, the data are compared to deter mine how well they

meet the criteria. This is done by comparing the individual MQOs to the acceptance criteria. Qualitative comparisons to acceptance criteria are, almost by definition, very subjective and involve the use of expert opinion. Inmany cases the expert opinion is that of the project manager or principal investigator, who may be subjected to unknown bias in evaluating how good the comparison is. The principal qualitative indicators are comparability and representativeness. For examples on diff

erent types of qualitative comparisons , help may be found in Chapter 3, S ection 3.4 , which discusses various forms of comparisons when considering integrating existing data into a project. Documentation of how the qualitative aspects of comparison of data to acceptance criteria have been met is essential. Appendix B contains supporting information on DQI s for which MQOs can be set as acceptance criteria for including existing data in a project. The DQIs that may be able to be quantitatively assessed are:
 precision;  bias;  accuracy

(the combination of precision and bias);  completeness; and  sensitivity
. A ty pical example would be to determine if the accuracy of the measuring instrument (precision plus bias, the D QIchosen) is at most 10 ppb (the MQO for this D Q I). The performance of the measuring instrument (actual data) would be used to determine if this MQO could be achieved. For quantitative comparisons, graphical and statistical anal ys e s can provide estimates of the strength of the comparison to criteria. The use of statistical analy ses when possible is recommended to minimize evaluation bias. The extent of statistical anal ys is is directly related to the wealth of information available about the project ; a poorly documented project being less useful than a well documented one. For this reason , a well documented QAPP should be developed to enable later researchers to evaluate the data for subsequent use. Reference to Data Quality Assessment: Statistical Methods for Practitioners, EPA QA/G
- 9S , ( EPA 2006b ) , a guidance document on statistical methods written for non
- statisticians, is recommended. 2.4.6 Interim Assessments of Data Quality Data quality is formall y assessed in the data review steps, which include data verification, data validation, and data usability . These steps generally don’t take place until samples are collectedQAPP Handbook 38 December 2012 Agency Review Draft and anal yz ed. However, there are many opportunities during a project to raise possible concerns with data quality . Possible indicators of data quality issues include (but are not limited to):
 observed field sampling practices that potentially impact data;  an anomalous event in the laboratory (e.g., fire or chemical spill) that could potentially impact data;  extreme weather (e.g., stream samples collected through a layer of ice); and  an intuitive sense that something isn’t right with how the project is proceeding. The QAPP should include statements encouragi ng project team personnel to document any concerns they

have over data quality

throughout the course of the project. The mechanism for making their concerns known, and the individual responsible for determining appropriate action should be identified. Inte rim assessments may also be combined with the verification and validation activities for the project (see Section 2.5.1) , and with any results from the comparisons to acceptance criteria (Section 2.4.5)
. 2.4. 7 Evaluation of Unconventional Measurements Some times there is a need to extend beyond standard anal ytical methods into unconventional methods. These can be thought of as any method for which there is not an already existing standard (e.g. EPA, ASTM, or Standard Methods) that has been widel y accepted an d validated. Unconventional methods should not be confused with modifications to an existing method. Unconventional methods may include both sampling (e.g. passive sampling for indoor air testing using thin films) and testing (e.g. three dimensional gas chromatography) procedures. Ty picall y they are developed to sample or measure an analy te that is new or for which a regulatory limit or threshold has changed. Many environmental forensic methodologies such as isotope ratio mass spectrometry , are also consi dered unconventional because they are not considered to be typical for measuring environmental contaminants
. Use of unconventional methods may or may not be allowed depending upon the environmental program under which the project is regulated. For some pro grams onl y methods that are codified are allowed and they

must be followed verbatim. If unconventional methods are allowed and appropriate for use on the project, the QAPP should include scientificall y valid and legall y defensible documentation of the proc edures relating to those methods. As with conventional methods, the quality of environmental data obtained via unconventional methods should support the intended use for the project, and ensure that the DQOs are met. Clear MQOs should be established and d ocumented in the QAPP that describe the level of precision, bias, and sensitivity

expected and needed to meet project objectives. Evaluation of these unconventional methods should occur during project assessments, and should be clearl y documented in the QA Reports and/or final report. Itmust be remembered, however, that research projects often use new methods for which performance data do not exist. A full documentation of such methods’ performance may be premature as these methods are amongst the first be ing developed. Evaluation should include review of at least these items:QAPP Handbook 39 December 2012 Agency Review Draft  demonstration that the method can be implemented as described in the QAPP;  documentation that records of the implementation method are created and maintained;  how data generated via this method achieves the MQOs set forth in the QAPP; and  how the data will be us ed for their intended purpose. Unconventional measurements come under greater scrutiny than standard measurements and so the level of quality assurance should be commensurate with that added scrutiny
. Note that the evaluation of unconventional methods should not occur during assessments by independent assessors. This activity should have already

been performed by

project personnel at an earlier stage in the development of the project and documented for review b y independent assessors. 2.4. 8 Evaluation of Unconventional Monitoring Projects Environmental projects span a very wide range of activities and there are some unusual situations where the standard methods for monitoring environmental conditions do not fully meet project needs. Some examples for which nontraditional environmental monitoring might be required include crab counts in the Chesapeake Bay , fish kills in L ouisiana rivers, environmental sampling for chemical agent s surrounding chemical weapons disposal facilities, or monitoring of radioactivity

levels in pools containing spent nuclear fuel rods. As with unconventional laboratory methods, it is especially

important that these nontraditional monitoring methods be wel l documented in the QAPP. Along with full descriptions of the methods, project
- specific MQOs should be stated for representativeness, completeness, and for comparability

if these data will be combined for use with any other environmental data. For some mon itoring projects, MQOs for precision, bias, and sensitivity may also be appropriate (e.g., the sensitivity

of carbon
- dating methods for estimating tree age). Evaluation of nontraditional monitoring methods should include an assessment implementation of th e methods against their SOPs or other description in the QAPP. The efficacy of the methods should also be evaluated against the MQOs stated in the QAPP. Finally , the usability of the data generated from any nontraditional methods should be evaluated to det ermine if i t i s of appropriate quality

to support its intended use on the project. **2 .5**  **REVIEW ,**  **EVALUATION OF USABIL ITY**  **, AND REPORTING**  **R EQUIREMENTS**  **(A CT**  **)** The final part of the paradigm concerns the actual data; are they valid and can they be used for the intended purpose. Verification and validation is followed by an evaluation of usability. Included in the usability section is a small discussion of statistical significance. This section offers guidance on Standards Clause 7. 9, Data Review, Verification an d Validation, and Data Usability Reporting, Standards Clause 7.10, Documents and Records Management, and Annex B 3.6 , Review, Evaluation of Usability, and Reporting Requirements. The elements in this section address project checks to see if the data or prod uct obtained will conform to the project’s objectives and to estimate the effect of any deviations
. These activitiesQAPP Handbook 40 December 2012 Agency Review Draft may be performed throughout the project to inform mid
- course corrections and /or at the completion of the project. Three distinct data evaluation steps are used to ensure that project data quality targets are met and should be applied to all data collected and used in environmental projects
. These steps appl y to all aspects of data production, including field sampling and anal ytical activities
. Although the data verification/validation/usability evaluation process outlined in the following sections is portrayed as a sequential process, it may be beneficial (and more cost effective) for many projects to combine steps
. Fo r example, the entity conducting the verification could also conduct the first step of the validation process concurrentl y . The method by which an organization i nspects, reviews, verifies, and validates data should be stated in the QAPP. Data verification entails confirmation b y examination and provision of objective evidence that the validated information fulfils specified requirements (sampling and analy tical) , or requirements mandated by a contract or agreement . Data validation means confirmation by examination and provision of objective evidence that the particular requirement for which the data or information was collected is fulfilled. It includes the process of checking whether the information meets the project’s specifications. Data usability is the determination of the adequacy of data, based on the results of v erification and validation meet the QAPP criteria. See Appendix D for a more complete discussion of the steps involved in data verification, data validation, and data usability
. 2.5.1 Dat a Verification and Validation Targets and Methods This section describes what information should be included in the QAPP and presents procedures for implementing the first two of the three data evaluation steps: verification and validation
. T o perform the data evaluation steps described above, reported anal ytical data should be supported by a complete set of auxiliary (or meta) data as defined in the QAPP
. These auxiliary d ata include sample receipt and tracking information, chain
- of
- custody records, tabul ated data summary forms, and raw anal ytical data for all field samples, standards, QC samples, and all other project
- specific documents that are generated
. If relevant raw data or sample information are not available or adequate to document data quality , then data anal ys i s becomes problematical. Lacking a clear linkage of auxiliary data to the project’s objectives creates a potential weakness in assessing the quality of the data and the inferences to be drawn from them. Verification is performed at the onset of the data review process to determine if the required information (the complete set of data) is available for further review
. It involves a review of objective evidence that validated information fulfills specified requirements
. The QAPP should contain procedures for how verification of field and anal ytic activities will be conducted. Validation occurs after verification and ensures that the data and information used in decision making are of appropriate quality for their intended use
. Validation guidance and documents may be attached or referenced in the QAPP and should address the following:QAPP Handbook 41 December 2012 Agency Review Draft  the process that will be used to validate sample collection, handling, field analy sis, and anal ytical laboratory

project data;  the specific validation proces s that will be used for each anal ytical group, matrix, and concentration level;  the procedures and criteria used to validate data information operations, which may include : the electronic or manual transfer, entry , use, and reporting of data for computer m odels, algorithms, and databases; correlation studies; and data plotting;  how validation of field sampling, handling, and analy sis activities will be documented such as QC signatures in field logs and QC checklist s ;  the person , identified b y title, respons ible for data validation. In some cases the degree of validation is dependent on the organization, program, contract, or grantee and consultation with the organization’s QA Manager is advised. The data validation process is constrained b y a number of fact ors, including contract requirements and client/management expectations
. However, in most cases there remains an opportunity

for the data validator or project team to exercise professional judgment in order to maximize the benefits of the data validation p rocess . For example, Contract Laboratory Program, National Functional Guidelines for Organic Data Review ( EPA 1999 ) includes a section titled “Overall Assessment,” describes the use of a brief narrative in which the reviewer expresses concerns and comments on the quality of the data. If the reviewer has access to the project’s MQOs, then the reviewer is encouraged to comment on the usability of the data
. 2.5.2 Quantitative and Qualitative Evaluations of Usability Using information provided by the data val idator, the project team considers whether data meet project quality

objectives or DQOs. The usability assessment is the final step of data review and can be performed only

on data of known quality (i.e., verified and validated data). Note that this differ s from comparisons to acceptance criteria in that here it is the project level objectives (DQOs) that are being considered, previously only the specific MQOs were considered. A typical example would be to see if the estimated mean level of contaminant (one of the project objectives) can be estimated to within 10 ppb (another project level objective) with 95% certainty

(the final project objective)
. To accomplish this step of data review, the project team should do the following and describe the results in the QAPP :
 s ummarize the usability assessment process and all usability assessment procedures, including interim steps , peer reviews, and any statistics, equations, and computer algorithms that will be used to assess data ;  d escribe the documentation that will be generated during usability assessment;  i dentify

the personnel (by

title and organizational affiliation) responsible for performing the usability

assessment;  d escribe how usability assessment results will be presented so that they identify trends, relationship s (correlations), anomalies; andQAPP Handbook 42 December 2012 Agency Review Draft  d escribe the evaluative procedures used to assess overall me asurement error associated with the project and include the MQOs described in Section 2 .2.6 . Statistical comparisons are very useful and are similar to those used with MQOs discussed in Section 2.2.6. EPA 2006b may be consulted for examples and instructions of ho w to perform statistical tests. For the success of any project, team members should be encouraged to contribute freel y in the overall evaluation of the project’s data. 2 .5. 3 Potential Li mitations on Data Interpretation The QAPP should address the action to be taken when the MQOs or project
- required measurement performance criteria are not achieved , if for various reasons , the project data are not usable to adequatel y address environmental questions (i.e., to determine if regulatory or project action limits have been exceeded) , or the data cannot be used to support project decision making. In addition to potential data usability issues , there are other considerations that might limit the use of the data within or beyond th e original project
. Is sues that might impact interpretation of data include (but are not limited to ) :
 unique qualities of the sample matrix;  inadequate metadata documentation (e.g., temperature, or barometric pressure);
 collection methods that alter the co mposition of the samples;  use of anal ytical methods which may nullify comparisons to standard analyses ;  differences in implied s ample support (e.g., different amount of media collected per sample , or samples representing different sized areas );  extent to which the planned statistical data collection method was compromised using judgmental samples; and  use of a j udgmental sampling scheme (which negates the ability to extrapolate results to a larger population of interest)
. The QAPP should state that any potential limitations of data interpretation should be documented in the metadata records associated with these data and included in the final project report. 2.5.4 Reconciliation with Project Requirements The QAPP should describe how results obtai ned from the project or task will be reconciled with the requirements defined by the data user or decision maker
. The proposed methods to analyze the data to identify

possible anomalies or departures from assumptions established in the planning phase of data collection should be included in the QAPP
. These should be determined prior to data collection to ensure they are driven by

project needs
. The QAPP should describe how reconciliation with user requirements will be documented, issues resolved, and how li mitations on the use of the data will be reported
. Specificall y, how sample collection, handling, and anal ys is procedures will be validated against the measurement performance criteria specified in Section 2 .2.6 should be described
. Also, the evaluative procedures to be used in validation toQAPP Handbook 43 December 2012 Agency Review Draft assess overall measurement error associated with the project, including attainment of the objectives defined in Section 2 .2.6 should be described. Considering how the data will be used to answer the main study questions is an important step in project planning, and should be documented in the QAPP
. Without consideration of this, it is possible to gather data according to plan and of high quality, but cannot be used to answer the questions at hand because no statistical m ethods are available to support the project needs
. The QAPP should describe how the data will be summarized or anal yz ed (e.g., qualitative anal ys is, and descriptive or inferential statistics) to meet project objectives
. If descriptive statistics are proposed, state , if appropriate , what tables, plots, and/or statistics (e.g., mean, median, standard error, or minimum and maximum values) will be used to summarize the data
. If an inferential method is proposed, indicate whether the method will be a hy pothesis test, confidence interval, or confidence limit , and describe how the method will be performed
. Specifically , the QAPP should briefly

discuss the five steps of the data quality assessment (DQA) process ( see EPA 2006 b ) which include :
 a review of the project’s objectives to assure that they are still applicable and a review of the sampling design and data collection documentation for consistency with the project objectives noting any potential discrepancies ;  a review of preliminary data from QA reports ( when possible) for the validation of data, calculation of basic statistics, and generat ion of graphs of the data to probe the structure of the data and to identify patterns, relationships, or potential anomalies ;  selection of the appropriate statistical procedures for summarizing and analyzing the data , including treatment of censored values (non
- detects) , based on the review of the performance and acceptance criteria associated with the project ’ s objectives, the sampling desi gn, and the preliminary data review, and identif

ication of the key underlying assumptions associated with the statistical tests;  verification that those underlying assumptions most likely hold, or whether departures could be acceptable, given the actual data and other information about the study ; and  determination of conclusions from the data b y p erforming the calculations pertinent to the statistical test, and document ation of the conclusions to be drawn as a result of these calculations
. If the design is to be used again, there should be an evaluation of the performance of the sampling design. Although the steps of the DQA will not be implemented until data collection is complete, the final DQA will have comparisons to both qualitative and quantitative criteria. These criteria should be documented in the QAPP. Quantitative (statistical) evaluations of the data can provide strong evidence in support of, or to refute, project hy potheses or prior assumptions
. Even if only qual itative comparisons are planned, semi
- quantitative methods such as graphical methods should be planned for in the QAPP. Statistical significance resulting from quantitative comparisons to planned criteria relies on the c oncept founded on evaluation that th e weight of evidence (data) supporting a h ypothesis is valid
. Itis never possible to have perfect knowledge about a studied population, but it is possible to learn enough about it to be able to say with confidence that a particular hy pothesis concerning that population cannot be true
. However, one should be very careful not to allow the statistics toQAPP Handbook 44 December 2012 Agency Review Draft dictate decisions without recourse to common sense
. In particular, as more and more data are collected, it becomes easier and easier to achieve statistical si gnificance
. The concern is that at some point it may

be possible to determine statistical significance at levels that are not of practical significance
. This can be illustrated through the following example: Based on operations at an industrial plant, and its waste - release permit, it is expected that the pH of water leaving the plant will be 5.9
. The releases are monitored by weekl y collections and each week these data are combined with all previous data and the average pH is compared to 5.9
. After the firs t few months, the average release pH is 5.88, which is not statistically

significantly

different from 5.9 and the conclusion of no real difference justified
. After several ye ars have elapsed , the average release pH is 5.8996 and this is statistically

signi ficantly

different from the permitted value of 5.9, but is a conclusion of a real difference justified? This is a case where having so much data allows the reviewer to identify very small differences from the expected level, but the statistically significant result may

very well not have any practical significance (in this case a difference in pH of 0.0004, which is barel y measurable)
. While statistics provide a strong and essential tool for environmental decision making, the science of statistics is not a substitute for common sense and can lead to bad decisions if not tempered with practicality
. This is particularly true with publications that tend to accept only papers that show a “statistically significant result” (often using 5% or even 1% as the level of significance)
. It is relatively rare that the publication editor has required the author to include a discussion of the practical meaning of the statistical significance using costs, benefits, and anal ys is of the magnitude of the observed effects. The QAPP should take care to specify the conditions under which statistical significance may not have practical meaning. It is worth noting that “statistical significance” should not be the sole indicator of importance of a result or conclusion. This was discu ssed b y the U.S. Supreme Court in Matrixx Initiatives, Inc v . Siracusano (March 22, 2011) ( Ziliak 2011). While the case involved security law, the defendant (Matrixx) tried to suggest that its line of demarcation on whether to release data was based on statistical significance. The Court disagreed unanimously . They said that the presence or absence of statistical significance is not the key factor as to whether an adverse effect is material. Justice Sotomayor wrote the opinion noting, “A lack of statistic ally significant data does not mean that medical experts have no reliable basis for inferring a causal link between a drug and adverse events”. Statistical significance is only part of the quantitative aspect in the interpretation of a project’s results. A lthough reconciliation with project requirements represents the final part of the quadripartite Plan
- Do - Check
- Act, it is important to emphasize the final results should be compared with the planned objectives of Section 2.2.6. Important deviation of result s obtained from those planned for can greatl y affect the effectiveness of the decisions or inferences made , and can allow the project team to make an inherentl y intuitive conclusion as to the overall validity of results. 2 .5. 5 Report s to Management The QAP P should i dentify the frequency and distribution of reports issued to inform management (EPA or otherwise) of project status, including reports on the results of performanc e evaluations and sy stem assessment s, results of periodic data quality assessments,QAPP Handbook 45 December 2012 Agency Review Draft and significant QA problems , and recommended solutions
. Assessment checklists, reports, requests for corrective action letters, and the corrective response letters should be included as attachments to or referenced in the QA management reports. Periodic QA management reports are intended to ensure that managers and stakeholders are kept informed of project status and results of all QA assessments
. Efficient communication of project status and problems will allow project managers to implement time ly and effective corrective actions so that the data produc ed can meet the project’s objectives
. The following issues may be included in QA management report s :
 s ummary

of project QA/QC program activities including tables, graphs, and charts;  a summary

of training conducted during the project;  s tatus of project and schedule delay s;  d eviations from the approved QAPP and approved amendments to the QAPP;  d escription and findings of TSAs and other assessments;  required corrective actions and implementation; and  l imitations on the use of measurement data generated. Although the se issues listed may be addressed in QA management reports, they should also be included in the QA/QC section of the final project report
. The final project report should, at a minimum, give a reco nciliation of project data with project objectives, a data summary (including tables, charts, and graphs), a s ummary of major problems encountered and their resolution , and any additional data quality concerns together with c onclusions and recommendations. Separate from reports to management are reports and publications subject to peer review, or pre
- dissemination review. Formal Agency Peer R eview is discussed further in Chapter 3, Section 3.4 and also in EPA 2006c. Pre
- dissemination review is discussed in Pre
Dissemination Review Guidelines (EPA 2006d).QAPP Handbook 46 December 2012 Agency Review Draft **C HAPTER**  **3 QAPP ELEMENTS FOR EVALUATING EXISTING DATA**  **3. 1**  **OVERVIEW OF QAPP ELE MENTS FOR EVALUATING**  **EXISTING DATA** In an increasing number of projects, recourse must be made to data other than new data produced specifically for this project. The use of existing data (data collected outside of this project) can be due to resource or timeliness restrictions, convenience , or may be the only potentially suitable data for consideration for this project. Agency Quality Policy CIO 2106.0 mandates the use of a QAPP: this Chapter gives advice on Standards Clause 7.5, QAPP, and Annex B.3 QAPP Elements and Requirements. Existing data are any data or information available to the project team originally collected for a purpose different from the one for which they are intended to be used within the project. They may be data collected by

the same project team previously for another p urpose, or they could be data from many ye ars in the past that just happen to relate to some aspect of the current project. For some programs the data were collected for a purpose similar to the proposed project but by a different organization. Existing data (sometimes erroneousl y known as “secondary data”, and occasionall y discussed as “secondary use of data”) are challenging to use because determining whether they

are of appropriate quality can be difficult to ascertain. Existing data are often used beca use they may be easily accessible, less resource
- intensive than collecting new data, available within a short time
- frame, or even collected for a similar project to the one the dataset is being considered for. S ome examples include data obtained from www.data.gov ; published literature, reports, and handbooks; state and local programs; outputs from existing models; pilot studies; GI S layers; survey s; and other Government agencies. Inspection of existing data for potential use can be greatl y helped b y examining the metadata that accompanies the dataset. Metadata is the information that describes the dataset and its quality criteria. The ultimate success of an environmental program or project depends on the quality of the environmental data used in decision making, and this quality depends significantly

on the ability to inspect the metadata and ascertain its applicability to the project. It will be rare that the existing data set will be exactly what is needed for the proposed project, and documentation of the difference between project requirements and existing data characteristics is essential. It is important to utilize the graded approach which allows the existing data set to be c onsidered in its entirety rather than by

individual characteristic unless the objectives of the project demand it. Probably the most important part of the QAPP is in the planning for obtaining the data or information because an error at this stage may be difficult to rectify

once the project is underway . One of the key outputs of the s ys tematic planning process is the construction of a conceptual model that describes the scientific and engineering process under investigation. The conceptual model is an i mportant tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. Further details on sy stematic planning are to be found in Annex C of the Standards and EPA 2006a.QAPP Handbook 47 December 2012 Agency Review Draft QAPPs must, at a minimum, address all elements required b y the Standards and these elements are described later in more detail with guidance for how to implement them. The QAPP is invaluable in not only

documenting all aspects of the project but serves as a structured resource for writing the final report on the project. The elements arranged according to the Plan
- Do - Check
- Act project life cycle, are described in the following sections
. **3. 2**  **PROJECT MANAGEMENT (P**  **LAN )**  Planning, the first part of the construction of a QAPP, deals mainly with the management aspects of the project. Most of the structure is simple good management practices, the only QA
related part being the establishment o f data or project quality objectives and measurement performance criteria (section 3 .2. 6 ). The section offers guidance on Standards Clause 7.5, QAPP, Standards Clause 7.7, Systematic Planning, Standards Clause 7.10, Documents and Records Management, and An nex B3.3 Project Management, and Annex B3.4 Data Acquisition. The elements in this section address the format and disposition of the QAPP, project a dministrative functions , project information , and goals. These elements document the backbone of the projec t - planning process and lay the groundwork for the more technical elements. The QAPP must describe the project adequatel y and the elements that can address the basic project management and objectives of the work include:
 title, version number, and approval/sign
- off sheets;  document format and table of contents;  distribution list;  project organization and schedule;  project/problem background and description;  data or project quality objectives and measurement performance criteria;  special training requirements/certification; and  documentation and records requirements. 3. 2.1 Title, Version , and Approval/ Sign
- Off Each QAPP should include a page with the title of the project and the name of the organizations involved in various aspects of that proje ct . The version of the QAPP should also be clearl y identified along with the title
. Itis acceptable to create separate title pages and signature pages, as long as the document title, version number, and date appear on the signature page. The names, titles , signatures, and signature dates of those approving the plan are also placed on this page. Individuals responsible for approving the QAPP may include the organization’s Technical Project Manager and QA Manager, and the EPA (or other funding agency) Projec t Manager and QA Manager . Their signatures indicate both their approval of the plan and commitment to follow the procedures noted. Other key personnel that may

sign the plan are other QA officers, prime contractors, and subcontractors.QAPP Handbook 48 December 2012 Agency Review Draft This approval infor

mation is ty

picall y the first page of the QAPP. Depending on the organization’s administrative policy , QAPP approval could also be in a separate memorandum. The signature dates indicate the earliest date when the environmental data operations portion of th e project can start (i.e., its effective date). The QA Manager will also determine if digital (electronic) signatures are acceptable for the approval of the QAPP. In addition to the title, version number, and approval signatures, it is important to include a revision history . Each time the QAPP is revised, as approved b y the QA Manager, the version number should be updated and the revision history should be amended to include a brief summary of the change and date. 3. 2.2 Document Format and Table of Contents The QAPP should be organized such that it meets the project’s needs, can be reviewed efficientl y, and meets the document control requirements of the QMS under which it is developed. A document control format, such as the example shown in Figure 5 , may b e used to support QAPP development, or a foot er created for each page to show revision status.

Figure 5 . D ocument control format example The Table of Contents will generall y list QAPP elements, as well as any tables, figures, reference s ection , and appendices necessary to th e project. If the QAPP author prefers to organize the plan differentl y than how the elements are organized in this Handbook a table may be inserted here to cross - reference where the information for each element may be found to simplify review. Depending on the type of project, anal ytical research protocols, or data management procedures may be attached. If the existing data are unaccompanied by SOPs , resort to other indicators of quality may have to be made and documented
. In the case where proprietary stan dards (for example, those of ASTM or I S O ) are used, reference to the location on the relevant website may be sufficient. 3. 2.3 Distribution Li st The distribution list identifies all individuals who should get a cop y of the approved QAPP, either in hard cop y or electronic format, as well as any subsequent revisions. Key personnel responsible for project implementation and funding, and who should have the currently approved QAPP, should be listed with their project titles or positions, organization names, email addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will need access to it, the distribution list also serves as an eas y reference of who needs to be alerted and provided with a revised version of the QAPP in the case that modifications are necessary
. Some organizations choose to provide the distribution list on the title or approval page, others elect to include this list in the project organization section when listing key personnel and their contact in formation. Project Name/# __________________

Section # _______________________ Revision # ______________________ Date ___________________________ Page ___________ of

____________QAPP Handbook 49 December 2012 Agency Review Draft 3. 2.4 Project Organization and Schedule It is important that roles and responsibilities are well defined prior to initiating project activities. Those individuals involved with the major aspects of the project are named in this element along with their project responsibilities. For example, the people responsible for maintaining, updating, and overseeing implementation of the QAPP would be named here. The personnel included in this element should include the lead scientists, researchers, modelers , consultants, and contact information for the external organization holding the existing data . If actual personnel cannot be initially identified, then the position description of that person’s function should be given. An organizational chart or table can be very helpful, and should be included if appropriate. It is also helpful to indicate lines of communication among individuals or groups, and this can be shown easily

on an organizational chart or an organizational network diagram. While a single indi vidual may

have more than one responsibility in a project, the project should be organized such that any person having QA responsibilities is independent of those generating and using project information. If this is not possible, an alternative method of ensuring effective QA review should be specified in the QAPP. The level of detail included in the schedule is left to the discretion of the QAPP authors. It may be beneficial to have a very detailed and strongl y s tated schedule for the project to follow. In this case, there is a risk of requiring QAPP revisions if the schedule needs to be changed during the lifetime of the project unless the QAPP specifically states it will not be revised simply due to schedule delay s . It is useful to include critical points in the project such as expected date of QAPP approval, sub
- section start and end points, or dates modeling subroutines need to be completed. When creating the schedule, allowance should be made for potential delay s and general inefficiencies inherent in any project. If the project includes regulatory or court
mandated deadlines, these should be highlighted to ensure their importance is noted. For projects in which deadlines or milestones are not well defined , a more generalized work schedule can be formul ated. 3. 2.5 Project Background, Overview, and Intended Use of Data This overview should give the reader an understanding of the problem to be solved, along with any pertinent background information for the project. It describes why the project will be done and what goals the project intends to accomplish. The general project goals stated here form the foundation for the entire study . Equall y important, the development and documentation of this element will help ensure that all project team members clearl y understand and agree on the underlying purpose of the project, increasing the likelihood that the project design will address and accomplish that purpose. In addition to the general overview, include information on the background and use of previously appro ved QAPPs relevant to this project. In addition to information about the project, it is good practice to include an outline of information that is currentl y not sufficiently developed. Clearly state who needs the information and what the intended use of this information will be. Problems that are more complex will lead to more extensive information in this section. The reader should be able to understand the importance or context of the project. The general project goals stated in this section will be refi ned in S ection 3.2.6.QAPP Handbook 50 December 2012 Agency Review Draft 3. 2.6 Data/Project Quality Objectives and Measurement Performance Criteria In this element, the QAPP should indicate how good the data or information must be to answer the question, resolve the problem, or support the decision to the level desired for this project
. For projects that use existing environmental data, acceptance or performance criteria for the data should be determined during the project planning process and documented in the QAPP
. EPA requires the use of sy stematic planning (see also Annex C of the two Standards) for all projects that involve the collection or use of environmental data and the QAPP documents the outcomes of this process. When using existing data it is very important that a sy stematic planning proces s has been used as it directl y influences the quality and representativeness of the data generated. EPA encourages the use of the DQO process, which may also be called project quality objectives, depending on the organization’s preference. The DQOs are est ablished from total study

variability

as the overall qualitative and quantitative goals of the project. Under any s ys tematic planning process for a project that includes environmental data collection it is desirable to define tolerable levels of uncertainty for components of the total study variability . Total study

variability

is due to natural (field) variability and measurement (laboratory) variability. The effects of field variability depend on the selecting of an appropriate sampling design and number of samples collected (see Appendix B for further discussion). Measurement variability

depends on the specific measurement and anal ytical techniques (see Appendix C for further discussion). DQOs are supported b y Data Quality Indicators (DQIs), which are su pported by

Measurement Quality Objectives (MQOs). These DQIs are linked to field and measurement variability (see Appendix B for further discussion). MQOs are sometimes known as Measurement Performance Criteria (MPCs) depending on the protocols of the orga nization. See Appendix B for definitions and details of DQI s and MQOs. It is essential that the definition of what is meant by MQOs or DQOs is made clear early in this section. Some organizations use the same acronym s for different operations. For example, some organizations use DQOs in place of DQI performance measures, or use MPCs for these measures. The traditional DQIs considered in project planning include:
 precision;  bias;  representativeness;  comparability ;  completeness; and  sensitivity
. While historically

there has been considerable attention directed to bias, precision, and sensitivity, it is really

representativeness that is probably the single most important indicator of data quality . Representativeness is a qualitative measure of the degree to which data accuratel y and precisely

represent a characteristic of a population.QAPP Handbook 51 December 2012 Agency Review Draft A poor sampling design with very high quality

analy tical laboratory

anal ys es might give data that meet MQOs for precision, bias, sensitivity, comparability , and comp leteness while not providing information that is actually representative of the characteristics of the population of interest. If the existing data fail to be representative of what was intended to be sampled, no amount of statistical manipulation can make the data set valid. Great attention should be given to ensure that the existing data are trul y represent ative of the project’s population of interest. It is axiomatic that a sufficient number of samples will be needed to assure a good representation of th e population has been achieved
. The QAPP should indicate how rigorous the existing data or information has to be to answer the question, resolve the problem, or support the decision to the level desired and needed for the project. Information on acceptance or performance criteria should be determined during the project planning process and documented in the QAPP. Establishing DQOs and MQOs for exploratory projects (for example, research studies) where little or no information on the quantitative magnitude of what is expected is difficult. The use of information from similar or related studies is useful. A discussion documented in the QAPP of the consequences (or results) of using existing data with unknown MQOs is advisable. 3 .2.7 Special Training Requirements and Certification It is not likely

that special training will be required when using existing data, but it is still possible (e.g., handling of C onfidential B usi ness Information, or security clearance)
. The QAPP should also specify

how this information will be documented, where the records will be kept, and indicate who is responsible for ensuring that these special training and certificate needs are met and documented. In some cases, reference to an outside authority or consultant with the skills and training appropriate for the evaluation of a specific scientific, engineering, modeling, or other technical area may be necessary . The name of the consultant together with a summary of their field of expertise should be documented in the QAPP. 3. 2.8 Docum entation and Records Requirements The QAPP should describe the process and responsibilities for ensuring that project personnel will receive the most recently approved project documents such as the QAPP, SOPs, and other documents used throughout the projec t operation
. QAPP authors should discuss how these documents will be updated and how information regarding updates will be communicated
. The QAPP should indicate where all project documents such as the QAPP or final report and records such as statutory

re quests and training records will be stored and for how long
. In some cases retention, access to, and final disposition of some records may be regulated
. In those cases, this element should address and compl y with all relevant regulations.QAPP Handbook 52 December 2012 Agency Review Draft **3. 3**  **DATA ACQUI SITION**  **(D O**  **)** This is the heart of a QAPP as it documents the requirements for dataset identification and access, the establishment of acceptance criteria, information on how the data were collected, and how the integrity of the data are assured. This sectio n offers advice on the Standards Clause 2.1, Scope, Clause 7.7, Systematic Planning, and Annex B3.4, Data Acquisition. 3. 3.1 Proposed Data Source Originator and Publication Information Knowing who originated ( initially collected , who generated, or was responsible for) the existing data can be very

informative
. Data originators might be EPA, s tates, tribes, municipalities, potentially

responsible parties (PRPs), or activist groups, for example
. Confidence in the data may , to some small extent, be judged according to the data originator
. That is not to say that data collected b y s tates, for example, should alway s be accepted, or that data collected by PRPs, for example, should alway s be excluded
. However, knowing who the data originator was may offer some information as to the purpose of the data collection and reflect on the possible impartiality of the lead author or source of the data
. Whether data are private or publicly accessible, and if they

have ever been published, may also be helpful to know
. Conv ersel y, if the data are identified without any understanding of who originall y collected them or fo r what purpose, that would be cause for very careful consideration of their appropriateness for use in the project
. The QAPP should discuss how the data source originators will be identified and documented, whether or not the data have previousl y been made public, and how any decisions regarding the use of data will be dependent on this information
. The rationale for th e selection of a particular database in preference to others should be discussed with respect to the acceptance criteria established for this project (see Section 3.3.3). If one dataset is considered “higher quality” than another , then the rationale for this decision must be documented in the QAPP . In some cases, the site where the dataset can be found is of importance when evaluating the suitability of the data for use. As an example, data available from a Government site would have been subjected to some set of quality assurance protocols (see section 3.4.1), whereas a site that does not practice or document their quality assurance protocols would be of lesser possible worth. 3. 3.2 Data Format and Accessibility The QAPP should describe, to the extent possible, how the existing data will be accessed and what format they

are in
. Data are stored in various ways ranging from national relational databases where access is relativel y eas y, to stacks of documents in the back corner of a dusty warehouse
. Any knowledge regarding data accessibility and format should be described in the QAPP, and if that is unknown, how it will be ascertained may be included
. Procedures for transferring the data to a format usable b y the project team should be discussed if relevant. For a further discussion see Brilis et al . 2004.QAPP Handbook 53 December 2012 Agency Review Draft 3. 3.3 Establishment of Acceptance Criteria Section 3. 2.6 dis cussed the need to document D QOs that apply to the project as a whole
. Acceptance criteria are the specific quality needs for existing data
. In many cases these may be universal (e.g., the same requirements may be applicable for all existing data brought into the project)
. There may

also be cases where the acceptance criteria for a particular set of existing data have slightly

different acceptance criteria than for other data brought into the project (e.g., the compiled data in full should be representative of the whole watershed, but th is particular data set is meant to be representative of only the area at the mouth of one specific creek)
. The QAPP should document the acceptance criteria for all existing data that will be included in the project. The use of assumptions to qualify the existing data set as adequate for use should be carefully documented as these can have an immense impact on the validity of the final project’s results if unknown bias is present. The use of assumptions such as “presumed to be valid”, or “assume this data can be used for this project” is severel y discouraged and effort should be made to establish objective acceptance criteria. It must be acknowledged that when considering existing data for potential use, quantitative acceptance criteria may

be difficult to construct. However, some concept of what the acceptable range of acceptance criteria needs t o be developed by the project team. Reliance on broad statements such as “we’ll use the best we can find” or “whatever is in the literature” should be discouraged. If the quantitative criteria are poorl y defined, then an iterative approach may be appropri ate where the characteristics of one existing data set are compared to those of another until the “best” is selected. It is advised that the acceptance criteria be phrased in a comparative fashion by

selecting an important DQI and then addressing the MQO f or that indicator. For example, “the dataset having the most precise estimate of variance will be selected”, the desired indicator, precision, is the acceptance criterion for the different datasets under consideration. When possible, the existing data should be compared to the MQOs (or MPCs depending on the nomenclature of the project’s organization) related to the objective of the project through examination of the metadata that accompanies each project . The principal DQIs (precision, bias, representativen ess, comparability , completeness, and sensitivity ) are especially

useful in establishing an overall set of descriptors of evaluators of existing datasets
. Other contributory indicators such as age of data, temporal representativeness, technological obsole sce should be developed if relevant and documented here. When using data from a variety of sources or sampling events, it is important to be sure that the data are in harmony (e.g., similar sample collection design, sample collection methodology , sample preparation, analy sis, and reporting) in the way s useful to the project’s objectives. When an existing dataset lacks information on MQOs, careful attention should be given to the overall DQOs of this dataset, because the set of MQOs directl y affect DQOs. A v ery “tight” achievement of the DQOs would be a good augury of the quality

of the dataset, a “looser” achievement, less so. Ingeneral, the QAPP should foster a “common sense” approach in the absence of the possibility of setting quantitative objectives and consequent reliance on qualitative objectives. When metadata are absent , recourse to statistical analy sis or comparisons to similar studies may be necessary

and then documented. In some circumstances, DQOs may be stated in qualitative terms having definitive points of reference.QAPP Handbook 54 December 2012 Agency Review Draft For example, “The existing data qualifies for the ‘A’ rating with respect to the rating s ys tem described in Section 4.4.2 of the Procedures for Preparing Emission Factor Documents (EPA
454/R
- 95 - 015)”. Although it is stated in quali tative terms, this DQO is measurable using specific acceptance criteria that are presented in the cited document. Simplistic descriptors such as “very good” or “acceptable for this project” are to be discouraged unless these are specificall y defined in th e QAPP. Qualitative acceptance criteria (for example, representativeness, or comparability) may be difficult to describe, but need to be couched in terms that clearl y indicate why one set of existing data may be preferred to another. For example, “the pro ject’s objectives require the cohort of data be exposed to defined environmental conditions for a period not exceeding 5 ye ars”, the selected indicator, comparability , being linked to the objectives of the project. 3. 3.4 Sample Data Collection Methodology The QAPP should give great attention as to how the existing data set has been collected with particular reference to whether a statistical (probabilistic) or judgmental data collection scheme was used. The inference or conclusions from the data hinge on this distinction. There can be a tremendous problem if judgmentally sampled data are misclassified as random (of some description) data as the probabilistic statements generated are unlikely to be valid. The same is true if data consisting of composite sampl es are regarded the same as if they were random samples. The problem is compounded when data from different collection or anal ys is are compared or evaluated against acceptance criteria. For example, are all data from the top six inches of soil, or are so me of the data from deeper intervals? Were all data collected according to statistically rigorous sampling designs such that inferences to the greater population can be made, or were some of the data collected based on best professional judgment? Do all data represent the same amount of environmental media, or are some sample s simple grabs while others are composite samples? These and many other similar questions shed light on the appropriateness of combining data and using them for a common purpose . The QAPP should identify and document what is important to the project team relating to sample data collection methodology
. 3. 3.5 Quality

Program and Quality

Assurance Procedures Used b y Data Originator If possible , the Q uality M anagement Plan (QMP) under which the existing data were collected, and the QA procedures that were used during data collection should be documented in the QAPP . At a minimum, the quality program should be referenced, and some documentation such as a QMP or text extracted from the QMP may be needed as an attachment to the QAPP to provide evidence that the data quality are expected to meet the project’s objectives
. When the quality

program under which the data were collected is unknown, discussion of how the project will deal with unknown data quality should be documented to avoid unacceptable uncertainties in project decisions that are based on existing data. If the quality program under which the data were collected is unknown when the QAPP is written, but is later identified, it should either be included in a revision to the QAPP or documented in the final report.QAPP Handbook 55 December 2012 Agency Review Draft It is, however, rare that existing data have the QC information readil y at hand although it is quite possible the QC procedures would be documented . The project team should carefully consider the importance of each type of information and determine how its absence will impact the project if this informatio n is not available. Data from EPA should have this information as part of the original QAPP as QC procedures are an integral part of the QAPP , but is less likely for other organizations. Effort should be made to ascertain the status of the data generator’ s QMP as it is possible that QC procedures would have been referenced as part of the analy tical SOW, thus giving more credence to the reliability of the data. Regardless of whether the quality program under which the data were collected is known, this section should include documentation of the project’s data quality requirements along with statements of how the data quality will be assessed. The overriding objective in considering the Quality Program under which the existing data was collected is ascertain ing the integrity of the data. It is important to ask the question “Can I trust these data?” together with the follow
- up question “What leads me to believe these data are reliable?” Documentation of qualitative (and quantitative if possible) aspects of th e dataset that answer these questions is essential in ensuring the dependability of these data. 3.3.6 Documentation of Sample Quality Assurance Procedures Section 3.3.5 discussed documentation of the QA procedures used when the existing data were originall y collected. This section is for QA procedures to be used as existing data are brought into the project. For example, if the existing data are available only in pdf files and need to be converted to spreadsheets for anal ys es, the procedures used to assure that errors in translation are minimized should be documented here. Some datasets may be subject to regulations such as the special rules governing certain information (40 CFR 3) and may have the needed information. In general, the QAPP should specify what procedures will be used to ensure that existing data maintain their original integrity and quality as they are migrated into current project files. For ex isting data to be used in the project, the QAPP should ideally identify the information to be included in the project data files (project record) and its format , if possible. 3. 3. 7 Data Management Requirements How the existing data will be stored , who will be responsible for access and maintenance, and h ow it will be incorporated with other project data are important elements to consider
. The element is meant to provide an opportunity to document the hardware, software, and personnel requirements for managing and incorporating existing data into the project. All aspects of data management, including handling, tracking , processing, and data transfer need QC
- ty pe measures to ensure the i ntegrity and dependability of the data is not lost. This would apply equally to both manual and electronic information sy stems and transfers between them
. The security of the database is one QC
- ty pe measure that helps maintain the integrity of the data. Access to the database with the ability to edit the data is a potential threat to integrity and the measures taken to prevent unauthorized or accidental editing of data should be documented. Project teams are also encouraged to retain information about the original sources of the data in the actual database so this information is available for future anal ys is.QAPP Handbook 56 December 2012 Agency Review Draft **3. 4**  **ASSESSMENTS (**  **C HECK**  **)** The third part of the Plan
-Do
- Check
- Act paradigm considers the qualitative and quantitative aspects of QA that apply to ensu ring the integrity of the existing data. The techniques used for generating new data rarely apply as existing data rarely are accompanied by detailed QC information. W

hen making assessment on existing data attention has to be given to careful use of the graded approach that selects the level of assessment commensurate with the intended use of the data. In some cases a “judgment call” will have to be made as the existing data may not have the detail desired to enable a good assessment to be made; documentati on of the rationale for making the decision to use the dataset is essential. The advice given here is drawn from the Standards Clause 7.8, Assessment of Data and Information, and Annex B3.5 Assessments. For existing data, technical sy stems audits, performance audits of measurement and anal ytical sy stems, surveillance of operations, and audits of data quality usually cannot be conducted unless QC information is avail able
. 3. 4. 1 Qualitative Comparisons to Acceptance Criteria Qualitative comparisons to acceptance criteria are, almost by definition, very

subjective and involve the use of expert opinion. Inmany cases the expert opinion is that of the project manager or principal investigator, who se judgment may be biased . Documentation of how the qualitative aspects of comparison of data to acceptance criteria have been met is essential. The two principal DQIs, comparability

and representativeness, are the most important indicators to consider. The following table shows examples of various aspects of quali tative comparisons: Table 1. Examples of Qualitative Comparisons Example In dicator of Comparability Example Questions and Answers for Comparison to Qualitative Criteria Samples within datasets should be selected in a similar manner Are the sample collection designs or methods compatible? If a

probabilistic sample design has been used, ensure they are the same type (e.g. random). If stratification has been used make sure the boundaries of the strata (sub
- populations) are comparable. I f a complex design has been used, assistance of a statistician may be necessary. Were the sampling locations or frequencies selected in a similar manner? If non
-

probabilistic (directed, or non
- random) samples were taken then there needs to be a justificati on of why these samples are acceptable for use in the project. Caveats may be necessary as the conclusions from non
-

probabilistic are difficult to extrapolate to wider situations of even populations of interest. If the non
- random samples are arguably similar to what would have been selected if a probabilistic sample had been used then a justification should be made. If a probabilistic sample has been augmented by non
- random samples, a minor justification should be made if the augmentation is small, but a mo re comprehensive justification would be needed if the augmentation is relatively large.QAPP Handbook 57 December 2012 Agency Review Draft Example In dicator of Comparability Example Questions and Answers for Comparison to Qualitative Criteria The samples should apply to the same target population Are they equally representative of the population of interest? Attention must be given to ensure the two sets o f samples really apply to the same target population; consideration of the metadata contained in the description of how the samples were obtained is essential. It is advisable to evaluate any differences with respect to potential bias in the conclusions drawn from comparisons of the two datasets or conclusions from combining the two datasets
. Data should be temporally and spatially consistent Were samples collected in the same sampling event? If there are

discrepancies in time, justification of why these time periods are of little importance will be needed. In some cases it may be necessary to argue these time periods are essentially “typical” of the situation. Are there temporal factors such as seasonality? This may be of importance and a justification of why this can be ignored will be needed. For example, comparing stream flow data from the winter season with that of the summer season may not make sense due to the different seasonal factors influencing stream flow
. Are the samples taken representative o f the same population? This may require a discussion on what representativeness really means in context with the project’s goals and objectives. There is no easy definition of representativeness and so justification using qualitative measures will be necessary. Were the samples in comparable matrices? The efficacy of extraction of contaminates from the sample obtained from the field is very dependent on how the contaminant binds or adsorbs to the inert materials of the matrix. Comparison of data obtained from a water sample with that from sediment may be problematical depending on the ease by which the contaminates can be measured. Reference to methods handbooks may be necessary to ascertain comparability. Datasets should contain the same set of variables of interest Were the variables of interest reported for all datasets ? Sometimes

measurements on key variable are not available. If only a few readings are missing it may be possible to justify imputation methods to substitute a value using information from auxiliary variables. This is particularly true when comparing or merging multivariate datasets
. It depends on what information is available from the metadata associated with the data set. For example, particle size, total organic carbon, or percent moistu re may be useful for determining if the data are comparable, but these variables may not always be reported.QAPP Handbook 58 December 2012 Agency Review Draft Example In dicator of Comparability Example Questions and Answers for Comparison to Qualitative Criteria Units in which these variables were measured should convert to a common metric The units of measurement should be reported for all datasets
. This goes

beyond a simple Imperial/Metric (ounces to grams) argument but to whether the units all convert to a common metric. For example, some results may be reported in wet weight and some in dry weight, which are not directly comparable without additional information. How this information is used becomes part of the justification. Field collection methods should be similar What instrument was used and which procedure was followed? It is

important to note the way in which the physical samples were collected as different methods may lead to different results. For example, documenting the actual collection method from a 6” core could be very different from that obtained from a composite sample of four 6’ cores that have been combined to make a single sample. Incremental samples (often between 50 and 200 very small samples combined into a single sample) should be examined with respect to the project objectives. The QA/QC criteria used in making the incremental sample must be documented to ensure the us e of this information matches the goal of the project. Inadvertent bias may result from the merging of data from differing sample collection methods and justification for use may have to be comprehensive. The same is true of comparing filtered samples with unfiltered samples, or where the sample handling conditions are markedly dissimilar. Rules for excluding certain types of observations should be similar for all datasets Are “outliers” treated in a similar fashion across all datasets? The

exclusion of data for being ‘too large’ or ‘too small’ is a potentially biasing situation. The reason for the exclusion of data must be carefully examined by the potential user as incorrect exclusion can create some extreme bias. The rationale for exclusion must be carefully considered with respect to the goals and objectives of the projec t. If no documented rationale exists, then a justification for its use is essential. The nomenclature should be consistent with the objectives of the project Are the definitions used in the existing dataset consistent with those of

the project? Convertible units (e.g. Imperial to Metric or  g/Kg to ppb

) are not the problem but definitions involving the target population and sample technique used could cause pro blems if too divergent. Metadata should be adequate to understand the protocols used Are the metadata (information about the way the actual data were obtained) adequatel y described? The metadata should describe how the samples (data) were collected and b e comprehensive enough for a reviewer to understand the purposes and procedures involved.QAPP Handbook 59 December 2012 Agency Review Draft Example In dicator of Comparability Example Questions and Answers for Comparison to Qualitative Criteria Similar analytical methods or procedures should be used to analyze the samples for all datasets. Was the same anal ytical method used for all sample results considered

and , if not , are the anal yt ical methods comparable? The use of routine methods simplifies the determination of comparability because all laboratories used the same standardized procedures and reporting parameters. However, when reviewing the analytical methods used to produce the data, consideration should also be given to options in methods that may have been used by the laboratory. Although the analytical method may be the same, options such as matrix or concentration level will affect results reported. For difficult matrices, different methods are sometimes used to obtain measurements of different analytes. If different methods have been used, a justification as to why the data are suitable for the project may be necessary. The same is true for the preparation methods used before the actual sample was analyzed. For example, data resulting from a total digestion preparation when compared with data from a partial digestion preparation would not result in directly comparable results. Measuring devices used for both datasets should have approximately similar detection levels Are the detection or quantitation levels acceptable for use in the

proposed project? Combining datasets having different detection or quantitation levels can lead to difficulties in analytical interpretations. For example, data set A may have a minimum detection of 5.0ppm, but data set B a minimum detection of 2.0ppm. Justification will be necessary to describe how data occurring between 2.0 and 5.0 in data set B will be treated when both datasets are merged together. Qualitative evaluations of data having been through a peer review sy stem are important when only partial information is available about a data set
. Inmost cases existing data have been through some form of review and it follows that publication resulting from an incomplete evaluation of data used should be accompanied by a disclaimer. Given that existing data have cleared some pre
- requisite before release, th ere are four c ases to c onsider; data occurring in a peer - reviewed publication, data from other Government organizations, data from within the EPA, and data that have been accepted b y the courts. Peer Review for Publication In the case of papers, articles, and discussion pieces written for external publication, the peer review procedure used is determined b y the individual publication’s review policy. This can range from a simple acceptance b y an editor for conformance to accepted practices for that publication, to double
- blind re view with full documentation for extremely rigorous publications. It is usual for most publications to employ an anony mous reviewer (referee) s ys tem (the reviewer does know the identity

of the paper’s author, hence single
- blind review) with at least one le vel of review b y peers competent in the same field as the author.QAPP Handbook 60 December 2012 Agency Review Draft In m any scientific publications the authors make some statement on the QC procedures used to generate the data but rarely give specifics. Additionally , they onl y occasionally make available the original data to the peer reviewers. A summary of the data or the data summary statistics usually appears in the published article. In many cases this is not due to the author’s wish to prevent the release of data (although this could be true in the ca se of proprietary data or data deemed to be of a sensitive nature) , but an economic one based on publication printing and distribution costs. For a publication constrained b y internal policy not to exceed a certain size/distribution, this could be the over
- riding factor in causing lack of data availability . Some publications make the data accessible to other interested parties for testing for internal consistency

and reliability , but that it is not a universal practice. In some instances a request to the principal author can result in the release of data in sufficient detail that comparison to a set of acceptance criteria may be made. If however, the data fall under proprietary

information such as Confidential Business Information, even a request to the prin cipal author may

not result in the release of data in a useful format. Consideration and evaluation of whether the data meet the Acceptance Criteria established b y the potential user of the data must still be determined and documented. The caveat that a full investigation of data characteristics could not be done should accompany the evaluation. Data from other Federal Organizations When considering the use of data from another Federal agency or organization, there is a minimum set of peer review requirements established. Nearly all Federal organizations have to comply with specific peer review requirements before disseminating influential scientific information and data and make available relevant data
. These requirements were published b y the Off ice of Management and Budget (OMB) in a Peer Review Bulletin issued December, 2004 (OMB 2004) , and provides the definition of influential scientific information as:

“ S cientific information the agency reasonably

c an determine will have or does have a clear and substantial impact on important public policies or private sector decisions.
... ( T ) he term “influential” should be interpreted consistently

w ith OMB’s government - wide information quality guidelines and the information quality guidelines of the agency .” The Bulletin provides detailed peer review guidelines and applies a more stringent set of requirements for “highl y influential scientific assessments” , which are further defined as: “...an evaluation of a body o f scientific or technical knowledge that ty pical ly s ynthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information.” The difference from a traditional peer review for publication is that there is public disclo sure of peer reviewers’ identities when dealing with highly influential scientific assessments. Inaddition it is required that the peer reviewers, their reports, and the response of the agency to those reports be made available to the public. The evaluation of whether the data meet the acceptance criteria established by the potential user of the data is made easier when peer review reports can be considered with the published dataQAPP Handbook 61 December 2012 Agency Review Draft and information. The decision of whether the data meet the definition of “i nfluential” (thus triggering the peer review requirements) has to be documented and use of data not defined as influential justified. Data from EPA The peer review requirements for Agency data are laid out in the Peer Review Handbook ( EPA 2006 c )
. It follo ws the guidelines established by the OMB Bulletin and requires that influential scientific information be peer
- reviewed in accordance with the handbook . The handbook provides a flowchart for conducting a peer review and includes a checklist for managers pl anning a peer review. The handbook notes “Peer review enhances the credibility and acceptance of the decision based on the work product” and encourages the widespread use of peer review
. According to Stephen L. Johnson, EPA Administrator in 2006 , “Peer review of all scientific and technical information that is intended to inform or support Agency decisions is encouraged and expected” ( EPA 2006c )
. In formation and data generated b y the Agency should be easier to evaluate against the acceptance criteria established by the potential user of the data. Informal collaboration or assistance from the data generator is often available , leading to a more comprehensive evaluation for potential use. A request for information in the form of the QAPP used to colle ct the data is often available from the data generator and is invaluable in documenting the suitability of the data for project use. The encouragement by the Administrator to apply peer review to a wider set of information increases the potential reliabili ty of the data. Data Accepted by the Courts; the Daubert Rule When considering data for a source involving a litigated court case, reference to the Daubert Rules of evidence should be made. Daubert is one of the first of several rulings by the United State s Supreme Court giving guidance on the admissibility of scientific theory

or evidence ( Brilis , Worthington, and Wait 2000 ) . It is usually used in conjunction with Rule 702 of the Federal Rules of Evidence. Scientific expert testimony (including data) submi tted for testimony first has to be admitted by the trial judge , who acts as a gatekeeper in assuring that the expert testimony trul y proceeds from documented scientific knowledge. The trial judge uses relevance and reliability (the Daubert factors) as well as Federal Rule 702 to make the determination to admit expert testimony as evidence. It follows that when data submitted for evidence are considered for possible use in another context, there is some sense that the data are reliable because they meet the Daubert and 702 Rules. The data does, however, needs to be evaluated against the acceptance criteria for the proposed project because data accepted for one purpose may not be suitable for another. It is axiomatic that basic QA principles involving the generation of the data have been achieved as it is not possible to meaningfull y test h ypotheses using unreliable data. Italso follows that the data should be reliable because they have been peer reviewed. The Daubert factors would also likely appl y to novel situations where a standard approach to performing the operation does not exist or is incomplete in document ation. Attention should be given to the overl yi ng charge that forms the basis for data inclusion for Agency use “to assureQAPP Handbook 62 December 2012 Agency Review Draft that Agency

decisions are supported b y data of adequate quantity, quality, objectivity, and utility for their intended purpose” ( see Clause 4 of the Internal Standard). Note that “intended purpose” is key to the establishment of clear objectives for the project, the associated mea surements, and their associated quality

control criteria. However, being accepted for evidence does not automatically imply they

are acceptable for other uses. The presiding judge has acted as a “gate
- keeper” in allowing th ese data as relevant and acceptab le for consideration for this particular case. Consideration and evaluation of whether the data meet the acceptance criteria established by the potential data user must still be determined and documented. 3. 4.2 Quantitative Comparisons to Acceptance Criteria The QAPP should describe how results obtained from the project or task will be reconciled with the requirements defined by the data user or decision
- maker
. The proposed methods to analyze the data to identify

possible anomalies or departures from assumptions established in the planning phase of data collection should be included in the QAPP
. These should be determined prior to data collection to ensure they are driven by

project needs
. The QAPP should describe how reconciliation with user requirements w ill be documented, issues will be resolved, and how limitations on the use of the data will be reported to decision makers
. Appendix B contains supporting information on DQI s for which M QO s can be set as acceptance criteria for including existing data in a project
. The DQIs that may be able to be quantitatively assessed are:
 precision;  bias;  accuracy

(the combination of precision and bias);  completeness; and  sensitivity
. For each of these DQIs, the QAPP may include a plan for assessing project data against the acceptance criteria set during project planning and documented within the QAPP
. The discussion of comparison of project data to acceptance criteria may include the following aspects:
 when in the project life cycle these comparisons will occur;  who wil l be responsible for these comparisons;  how the comparisons to M QO s will be documented; and  what action will be taken if the project data do not meet the M QO s. Statistical comparisons are very useful and are discussed in EPA 2006b which has examples and in structions o n how to perform statistical tests. For the success of any project, team members should be encouraged to contribute freel y in the overall evaluation of the project’s data. There are, however, many instances where comparisons to acceptance cri teria for MQOs cannot be achieved.QAPP Handbook 63 December 2012 Agency Review Draft This is usually

because the published dataset onl y gives the final DQOs and not the specific MQOs. In such cases a note or caveat should accompany the description in the QAPP that this aspect of the reliability

of the d ata could not be fully confirmed. 3. 4. 3 Assessments to Other Criteria Be yond the DQIs and M QO s defined for the project, there may be other criteria against which the use of data should be assessed
. Examples of other aspects that may be included in such as sessments include the following:
 cost or resource constraints for incorporating existing data;  the timeframe in which the data will be available;  access to the phy sical site of study ; and  environmental justice concerns
. The timing of these ty

pes of assessm ents may

be mentioned in the QAPP (e.g., during readiness review, after the first data are incorporated, or at the conclusion of data analy s e s), as well as responsibility

for conducting them and the process for documenting their conclusions. 3.4.4 Interim Assessments of Data Quality It may be appropriate to conduct interim assessments as part of an iterative sequence of determining the suitability of existing data. The depth to which this can be done depends on the acceptance criteria established for the project. It is sometimes used to show why a certain data set should not be considered for a particular project. Possible indicators of data quality issues include:
 field sampling practices that potentially impact data quality

or reliability ;  anomalous events potentially influencing data quality ;  extreme weather conditions; and  an intuitive sense that something isn’t right with the data. Of course, the latter indicator is subjective and should be discussed with the entire project team. Its inclusion here is to act as a warning signal that further investigation into the background of the existing data is warranted. Interim assessments may also be combined with any results from the comparisons to acceptance criteria (Section 3.4.2). 3. 4. 5 Evaluation of Unconventional Measurements Unconventional methods can be thought of as any method for which there is not an already existing standard that h as been widel y accepted and validated
. If data or measurements from unc onventional methods are used for a project, the QAPP should include scientifically defensible documentation wh y these procedures are appropriate for this project
. The quality

of enviro nmental data obtained via unconventional methods should clearl y support the intended use for the project and, when possible, quantitative acceptance criteria for selected DQIs should be developed.QAPP Handbook 64 December 2012 Agency Review Draft If qualitative acceptance criteria are constructed, then a clear, detailed, explanation of why these measurements are used is needed. Use of data from unconventional methods will come under increased scrutiny by

reviewers, hence the need for detailed explanations. 3. 4. 6 Evaluation of Unconventional Monitoring P rojects As with unconventional laboratory methods, it is especially

important that nontraditional monitoring methods be well documented in the QAPP
. Along with full descriptions of the methods, acceptance criteria for qualitative indicators will be needed. The us e of the data generated from any nontraditional methods can be difficult unless documentation of why the proposed data should be used is included in the QAPP
. **3. 5**  **REVIEW ,**  **EVALUATION OF USABIL ITY**  **, AND**  **REPORTING REQUIREMENTS (** **A CT**  **)** The final part of the paradigm concerns the actual data; are they valid and can they be used for the intended purpose. The evaluation of usability must include any perceived limitations on the use of the data and with the reconciliation with project requirements is a small discussion of statistical significance. This section offers guidance on Standards Clause 7.9, Data Review, Verification and Validation, and Data Usability Reporting, and Annex B3.6, Review, Evaluation of Usability, and Reporting Requirements. The elements in this section address project checks to see if the data or product obtained will conform to the project’s objectives and to estimate the effect of any deviations. These activities may be performed throughout the project to inform mid
- co urse corrections and/or at the completion of the project. 3. 5.1 Data Verification and Validation Targets and Methods For existing data it is not common for verification and validation details to be found and recourse has to be made to the provenance of the data. If the generator of the data has a known quality sy stem, there is some assurance that the data have been verified and validated. This is especiall y true for data generated by or for the EPA as the Agency ’s quality s ys tem has verification and validation as an integral part of the s ys tem. Data from organizations with less documented quality s ys tems may give sufficient cause for concern that caveats should be attached to any conclusions reached from the data. 3. 5.2 Quantitative and Qualitative Evaluations of Usability Usability of existing data depends on the objectives of the project and the extent to which the existing data meet quantitative and qualitative acceptance criteria. All existing data have use , but the strength of the conclusions reached b y anal ys is of the data varies directly with the representativeness of the data with respect to the projec t’s objectives. Based on information gleaned through qualitative and quantitative comparisons to acceptance criteria, and reconciliation with project requirements , the project team considers the degree to which the data meet project quality objectives . Thi s requires the careful discussion of where the data are inadequate or relativel y weak in desired quality . This might be as simple as a precision estimate, but could be as much as an unintended change of population coverage due to data collection methodolo gy .QAPP Handbook 65 December 2012 Agency Review Draft Note that this differs from comparisons to acceptance criteria in that here project level objectives are being considered, previously only the specific MQOs were considered. A ty pical example would be to see if the estimated mean level of contaminant (one of the project objectives) can be estimated to within 10 ppb (another project level objective) with 95% certainty (the final project objective). The use of statistical comparisons to quantitative acceptance criteria is encouraged , but with a warning that any degree of statistical significance must be tempered b y the extent to which the existing data deviates from a truly random sample. The statistical anal ys is can be used to “weigh the evidence” b y making comparative conclusions instead of actual lev els of significance. Data Quality Assessment: Statistical Methods for Practitioners, EPA QA/G
- 9S , (EPA 2006b) contains many commonly

used statistical procedures and has been written for the non
- statistician. Comparisons to qualitative acceptance criteria such as representativeness and comparability are by definition, subjective , and can be affected b y the unintentional bias of the project team. Qualifiers and caveats as to interpretation of the results are to be expected (see also Section 3.5.3). It is advised that a summary of the project team ’ s collective reasoning be included in the conclusions from the analy sis of the data. 3.5.3 Potential Limitations on Data Interpretation The QAPP should address the action to be taken when project
- required M QOs ( acceptance criteria) are not achieved or for other reasons existing data may not be immediately usable with respect to the project objectives. In addition to the potential data usability issues, there are other considerations that might limit the interpret ation of the data such as:
 information gaps regarding data collection methodology ;
 information gaps regarding QA procedures used during data collection;  use of nonstandard anal ysis methods;  undocumented gaps in supporting metadata;  inclusion of potentially anomalous data not documented as outliers;  sample support (e.g., different amount of media collected per sample , or samples representing different sized areas ); and  the use of a judgmental sampling design
. The QAPP should state that any potential limitations of data interpretation should be documented and included in the final project report and included in the metadata stored with the project data
. The use of footnotes and separate appendices to caveat weaknesses in interpretation of data is discouraged as these tend to be omitted when project reports are sent to outside parties. The potential limitations should be in the text of the summary report. 3. 5.4 Reconciliation with Project Requirements Considering how the data will be used to answer the main study questions is an important step in project planning, and should be documented in the QAPP
.QAPP Handbook 66 December 2012 Agency Review Draft Without consideration of this, it is possible to compile data of high quality that actuall y cannot be used to answer questions because no statistical methods are available to support the project needs
. The QAPP should describe how the data will be summarized or analyzed (e.g., qualitative anal ys is, descriptive , or inferential statistics) to meet project objectives
. If descriptive statistics are propose d, state what tables, plots, and/or statistics (e.g., mean, median, standard error, and minimum and maximum values) will be used to summarize the data
. If an inferential method is proposed, indicate whether the method will be a hy pothesis test, confidence interval, or confidence limit and describe how the method will be performed
. Specifically , the QAPP should briefl y discuss the five steps of the DQA process (see EPA 2006b):
 a review of the project’s objectives to assure that they are still applicable and a review of the sampling design and data collection documentation for consistency with the project objectives noting any potential discrepancies;  a preliminary

data review of QA reports (when possible) for the validation of data, calculation of basic stati stics, and generation of data graphs to learn about the data structure and to identify

patterns, relationships, or potential anomalies ;  selection of the appropriate statistical procedures for summarizing and analyzing the data, based on the review of the p erformance and acceptance criteria associated with the project ’ s objectives, the sampling design, the preliminary data review, and i dentification of the key underlying assumptions associated with the statistical tests;  verification that those underlying as sumptions most likely hold, or whether departures could be acceptable, given the actual data and other information about the study ; and  determination of conclusions from the data b y p erforming the calculations pertinent to the statistical test, and document ation of the conclusions to be drawn as a result of these calculations
. If the design is to be used again, there should be an evaluation of the performance of the sampling design. Although the steps of the DQA will not be implemented until data compilation is complete, planning for DQA including selecting likel y statistical methods for analyzing the data and understanding the underlying assumptions of those methods is vital to do prior to data compilation to avoid missteps. Quantitative (statisti cal) evaluations of the data can provide strong evidence in support of ( or to refute ) project hy potheses. When considering using published data for purposes other than what it was collected for, attention should be given to the practical significance other wise misinterpretation of importance may ensue. This is particularly true when considering the use of data concerning a popular area for research or study. It is possible that the significant result was published, while all the non
significant results were deemed not suitable for publication. Statistical significance is a concept based on the weight of evidence that a hypothesis is valid
. It is never possible to have perfect knowledge about a population being studied, but it is possible to learn enough abo ut it to be able to say with confidence that a particular hypothesis concerning that population cannot be true
. However, one should be very careful not to allow statistics to dictate decisions without recourse to common sense
. In particular, as more data are collected, it becomes easier to achieve statistical significance
.QAPP Handbook 67 December 2012 Agency Review Draft The concern is that at some point it may be possible to determine statistical significance at levels that are not of practical significance
. This can be illustrated through the following example: Based on operations at an industrial plant, and their waste
- release permit, it is expected that the pH of water leaving the plant will be 5.9
. The releases are monitored by weekl y collections and each week these data are combined with all previou s data and the average pH is compared to 5.9
. After the first few months, the average release pH is 5.88, which is not statistically

significantly

different from 5.9 and the conclusion of no real difference justified
. After several ye ars have elapsed the average release pH is 5.8996 and this is statistically

significantly

different from the permitted value of 5.9, but is a conclusion of a real difference justified? This is a case where having so much data allows the reviewer to identify very small differenc es from the expected level, but the statistically significant result may

very well not have any practical significance (in this case a difference in pH of 0.0004, which is barel y measurable)
. While statistics provide a strong and essential tool for enviro nmental decision making, the science of statistics is not a substitute for common sense and can lead to bad decisions if not tempered with practicality
. This is particularly true with publications that tend to accept only papers that show a “statistically significant result” (often using 5% or even 1% as the level of significance)
. It is relatively rare that the publication editor has required the author to include a discussion of the practical meaning of the statistical significance using costs, benefits, and anal ys is of the magnitude of the observed effects. The QAPP should take care to specify the conditions under which statistical significance may not have practical meaning. It is worth noting that “statistical significance” should not be the sole indicator of importance of a result or conclusion. This was discussed b y the U.S. Supreme Court in Matrixx Initiatives, Inc v . Siracusano (March 22, 2011) (Ziliak 2011). While the ca se involved security law, the defendant (Matrixx) tried to suggest that its line of demarcation on whether to release data was based on statistical significance. The Court disagreed unanimously . They said that the presence or absence of statistical signif icance is not the key factor as to whether an adverse effect is material. Justice Sotomayor wrote the opinion noting, “A lack of statistically significant data does not mean that medical experts have no reliable basis for inferring a causal link between a drug and adverse events”. Statistical significance is only part of the quantitative aspect in the interpretation of a project’s results. The QAPP should also identify data reporting requirements, including data reduction procedures specific to the project and applicable calculations and equations
. In addition, data storage requirements for both hard copy and electronic data, as well as software requirements for the anal ys is should be specified. 3. 5. 5 Reports to Management The QAPP should identify the frequency

and distribution of reports issued to inform management (EPA or otherwise) of the project status, including results of periodic data
- quality assessments, significant QA problems , and recommended solutions. Periodic QA management reports are intended to ensure that managers and stakeholders are kept informed of project status and results of all QA assessments.QAPP Handbook 68 December 2012 Agency Review Draft Efficient communication of project status and problems will allow project managers to implement timely

and effective corrective actions so that data produced for the project can meet the project’s objectives. The QAPP should describe the content of each QA management report to be generated for the project, including an evaluation of measurement error (when possible) as determined from the as sessments. Assessment checklists, reports, requests for corrective action letters, and the corrective response letters should be included as attachments to or referenced in the QA management reports. The following issues may be included in QA management reports:
 summary

of project QA program activities including tables, charts, graphs depicting results ;  status of project and schedule delay s;  deviations from the approved QAPP and approved amendments to the QAPP;  required corrective actions and implementation ; and  limitations on the use of measurement data generated. Although these issues listed may be addressed in QA management reports, they should also be included in the QA section of the final project report. The final project report should, at a minimum, g ive a reconciliation of project data with project objectives, data summary (including tables, charts, and graphs), a summary of major problems encountered and their resolution, and any additional data quality

concerns together with conclusions and recommen dations. As many Agency reports are being read by an ever increasing audience, it is recommended that sufficient detail and explanation is presented in the final report to allow for an independent assessment of whether the existing data used in the project is suitable for reaching the project’s objectives.QAPP Handbook 69 December 2012 Agency Review Draft **CHAPTER 4**  **QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, AND USE OF MODELS**  **4. 1**  **OVERVIEW OF QAPP ELE MENTS FOR MODELS**  The use of models is becoming an increasingly important part of the Agency’s investigation into environmental phenomena and Agency Quality Policy CIO 2106.0 mandates the use of a QAPP. The QAPP has some elements very similar to those for QAPPs for new or existing data, but there are important differences to allow effective model development and use. This Chapter offers advice on Standards Clause 7.6, Use of Information Technology Methods and Sources, and Annex B3.4, Data Acquisition. This chapter focuses on QAPPs for developing, modifying, appl yi ng, and evaluating models, and for project use of data from models. The QAPP, using the elements described in this chapter, should provide the project team with a clear path for determining if use of models will achieve the project objectives or DQOs. Probably the most important part of the QAPP is in the planning for obtaining the data or information because an error at this stage may be difficult to rectify

once the project is underway . One of the key outputs of the s ys tematic planning process is the construction of a conceptual model that describes the scientific and engineering process under investigation. The conceptual model is an important tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. In some modeling projects the conceptual model is the model itself, but in others the model being investigated may be onl y part of the complete conceptual model. Further details on sy stematic planning are to be found in Annex C of the Standards and EPA 2006a. The use of models adds many nuances to traditional EPA QAPPs. The QAPP is meant to provide a description of how the project team will ensure that the project attains a level of quality commensurate with its importance. The project team should document all aspects of the project that could impact project quality . The QAPP is not meant to be burdensome, and any sections of this chapter that do not pertain to a particular project involving the development, modification, or use of models should simply present why

those sections do not. The ultimate purposes of the QAPP are for the project team to ensure that using the data, model, or both result in achieving the project objectives, and to pro vide documentation and defensibility for the quality

of the project results. EPA uses a wide range of probabilistic, fate and transport, and other types of models to inform decisions that support its mission of protecting human health and safeguarding the natural environment

– air, water, and land

– upon which life depends
. These models include atmospheric and indoor air models, ground
- water and surface
- water models, multimedia models, chemical
equilibrium models, exposure models, toxicokinetic models, ris k
- assessment models, and economic models
. These models range from simple to complex and may employ a combination of scientific, economic, socio
- economic, or other ty pes of data. As defined in this guidance, a model i s an application tool, while a model framework is the sy stem of governing equations.QAPP Handbook 70 December 2012 Agency Review Draft As models become increasingl y significant in decision making, it is important that the model development , modification, use and evaluation processes conform to protocols or standards that help ensure the utility , documentation , and defensibility of the models and their outputs for decision making. EPA’s Council for Regulatory Environmental Modeling (CREM) is a valuable resource for modelers with Guidance on the Development, Evaluation, and Application of Environm ental Models (EPA 2009b ) and self
- study training modules at www.epa.gov/CREM
. The following figure (Figure 6 , The Modeling Process) [ EPA 2009b , page 61, figure D.1.1 ] shows model development, evaluation, and application, and the relationship between them, including the use of data, data quality assessment and peer review. Note the order in the figure, where evaluation is central to both development and application. The development and use of mathematical and e lectronic models (IT products) sh ould be described in a QAPP and meet the applicable requirements of the Internal and External Standards. Further information is provided in this chapter; the elements are described in more detail in this chapter and organiz ed by the life cycle of the project according to the Plan
- Do - Check
- Act paradigm. QAPPs should , at a minimum, address all elements required b y the Internal and External Standards as detailed in this guidance. Elements can be addressed with a simple statement of why the information is not relevant , as mentioned previously , or with a cross
- reference to another approved document in which the information appears . It is acceptable to add sections to a QAPP beyond those identified. For example, if a project involv es heav y use of computers run in parallel to accomplish runs of complex models, the computer requirements might be added as a separate element in the QAPP. If the project must first set objectives and define what is to be accomplished by the model, it may be most appropriate to develop the QAPP in a phased manner (see Chapter 1, Section 1.5). The first stage might include all of the project planning and project quality objectives, and the latter stages might include model development or model selection and setting a schedule for the appropriate assessments to ensure the model is operating as expected. If a phased approach is used, the project must not move beyond the activities addressed in the QAPP until the next phase of the QAPP is developed and approved
. This chapter focuses on QAPPs for modeling projects with the elements are arranged according to the Plan
- Do - Check
- Act project life cycle. **4.2 PROJECT MANAGEMENT (**  **P LAN**  **)** Planning, the first part of the construction of a QAPP, deals mainly with the management aspects of the project. Most of the structure is simple good management practices, the only QA
related part being the establishment of data or project quality objectives and measurement performance criteria (section 2.2.6). The section offers guidanc e on Standards Clause 7.5, QAPP, Standards Clause 7.7, Systematic Planning, Standards Clause 7.10, Documents and Records Management, Annex B3.3 Project Management, and Annex B3.4 Data Acquisition as it applies to models. The elements in this section address the format and disposition of the QAPP, project a dministrative functions , project information, and goals. These elements document the backbone of the project planning process and lay the groundwork for the more technical elements.QAPP Handbook 71 December 2012 Agency Review Draft **Figure 6**  **. The Modeling Process** QAPP Handbook 72 December 2012 Agency Review Draft The QAPP must describe the project adequatel y and the elements that can address the basic project management and objectives of the work include:
 title, version number, and approval/sign
- o ff sheets;  document format and table of contents;  distribution list;  project organization and schedule;  project/problem background and description;  data or project quality objectives and measurement performance criteria;  special training requirements/certification; and  documentation and records requirements. 4.2.1 Title, Version, and Approval/Sign
- Off Each QAPP should include a page with the title of the project and the name of the organizations involved in various aspects of that proje ct. The version of the QAPP should also be clearl y identified along with the title. It is acceptable to create separate title pages and signature pages, as long as the document title, version number, and date appear on the signature page. The names, titles , signatures, and signature dates of those approving the plan are also placed on this page. Individuals responsible for approving the QAPP may include the organization’s Technical Project Manager and QA Manager, and the EPA Project Manager and QA Manager. Their signatures indicate both their approval of the plan and commitment to follow the procedures noted. Other key

personnel that may

sign the plan are upper management, other QA officers, prime contractors, and subcontractors. The QA Manager will also det ermine if digital (electronic) signatures are acceptable for the approval of the QAPP. This approval information is ty picall y the first page of the QAPP. Depending on the organization’s administrative policy, QAPP approval could also be in a separate memor andum. The signature dates indicate the earliest date when the environmental data operations portion of the project can start (i.e., its effective date). In addition to the title, version number, and approval signatures, it is important to include a revis ion history . Each time the QAPP is revised, as approved b y the QA Manager, the version number should be updated and the revision history should be amended to include a brief summary of the change and date. 4.2.2 Document Format and Table of Contents The Q APP should be organized such that it meets the project’s needs, can be reviewed efficientl y, and meets the document control requirements of the QMS under which it is developed. A document control format, such as the example shown in Figure 7 , may be used t o support QAPP development, or a footnote created for each page to show revision status.QAPP Handbook 73 December 2012 Agency Review Draft Figure 7 . Document control format example The Table of Contents will generall y list QAPP elements, as well as any tables, figures, reference s ection , and appendices necessary to th e project. If the QAPP author prefers to organize the plan differentl y than how the elements are organized in this Handbook a table may be inserted here to cross - reference where the information for each element may be found to simplify

review. SOPs may be included as appendices to the QAPP. Depending on the typ e of project, anal ytical research protocols, or data management procedures may be attached. If SOPs or other data gathering, data anal ys is, or evaluation protocols are not documented in, or attached to the QAPP, they must be available through some other means. The location of the SOPs referenced in the QAPP should be accessible to both QAPP users and QAPP reviewers. 4.2.3 Distribution List The distribution list identifies all individuals who should get a copy of the approved QAPP, either in hard cop y or electronic format, as well as any subsequent revisions. Key personnel responsible for project implementation and funding, and who should have the currently approved QAPP, should be listed with their proj ect titles or positions, organization names, email addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will need access to it, the distribution list also serves as an eas y reference of who needs to be alerted and provided with a revised version of the QAPP in the case that modifications are necessary
. Some organizations choose to provide the distribution list on the title or approval page, others elect to include this list in the project organization section wh en listing key personnel and their contact information. 4.2.4 Project Organization and Schedule It is important that roles and responsibilities are well defined prior to initiating project activities. Those individuals involved with the major aspects of th e project are named in this element along with their project responsibilities. For example, the staff responsible for maintaining, updating, and overseeing implementation of the QAPP would be named here. The personnel included in this element should includ e the lead scientists, researchers, modelers, consultants, QA personnel, and contact information for involved external organizations (contractors and sub
- contractors) . If the actual personnel cannot be initially identified, then the position description of that person’s function should be given. At a later date, after personnel have been identified, amendments to the QAPP may

be made (at the discretion of the QA Manager). An organizational chart or table can be very helpful, and should be included if approp riate. It is also helpful to indicate lines of communication among individuals or groups, and this can be shown easily

on an organizational chart or an organizational network diagram.

Project Name/# __________________ Section # _______________________ Revision # ______________________ Date ___________________________ Page ___________ of

____________QAPP Handbook 74 December 2012 Agency Review Draft While a single individual may have more than one responsibility in a project, the project should be organized such that any person having QA responsibilities is independent of those generating and using project information. If this is not possible, an acceptable alternative method of ensuring effective QA review should be spe cified in the QAPP. The level of detail included in the schedule is left to the discretion of the QAPP authors. It may be beneficial to have a very detailed and strongl y s tated schedule for the project to follow. This increases the a risk of requiring QAPP revisions if the schedule needs to be changed during the lifetime of the project unless the QAPP specifically indicates that such changes need not cause a formal revision of the QAPP. Itis useful to include critical points in the project such as expected date of QAPP approval, sub
- section start and end points, or dates modeling subroutines need to be completed. When creating the schedule, allowance should be made for potential delays and general inefficiencies inherent in any project. If the project includes regulatory or court
mandated deadlines, these should be highlighted to ensure their importance is noted. For projects in which deadlines or milestones are not well defined, a more generalized work schedule can be formulated. The project management elements of a QAPP for modeling activities are parallel to those for a QAPP for a project based on newly collected data or on existing data, as described in Chapters 2 and 3, respectivel y . These chapters can be helpful for projects combining activities. 4.2.5 Project Background, Overview, and Intended Use of the Model For modeling projects, more effort is put into this, the planning aspect of project development, than any of the other part s of Plan
- Do - Check
- Act paradigm. This stems from the fact that if insufficient attention is given to planning, potentially erroneous results may occur (a variation of “garbage
- in, garbage
- out”). This overview should give the reader an understanding of the problem to be solved, along with any pertinent background information for the project to provide a historical and scientific perspective for future projects. It describes why the project will be done and what goals the project intends to accomplish. The general project goals stated here form the foundation for the entire study . Equally

important, the development and documentation of this element will help to ensure that all project team members clearly understand and agree on the underlying purpose of the p roject, increasing the likelihood that the project design will address and accomplish that purpose. Included in this discussion should be a summary of any previously approved QAPPs relevant to the project or study . This should include both those specific to modeling and those involving the collection or use of data (see also section 4.2.6)
. 4.2.5.1 QAPP Requirements for Model Development/Modification This section expands on the demands of Standard B3.4.4.1, QAPP Requirements for Model Development/Modification and integrates specific advice with the Council for Regulatory Environmental Modeling “Guidance on the Development, Evaluation, and Application of Environmental Models”
. After the specification of the project objective and ty pe of model needed to address the problems associated with the project objective is the development of a suitable model framework.QAPP Handbook 75 December 2012 Agency Review Draft “A model framework is a formal mathematical specification of the concepts, procedures, and behaviors underlying the sy stem, object, or process relevant to the problem of interest, usuall y translated into computer software ” (EPA 2009b) . The structure of the model should be clearly stated especially

whether it is a stochastic model or a deterministic model as this is of crucial importance in the interpre tation of model output. A stochastic model (one founded on probabilistic principles involving random events or fluctuations) may result in an output where the results must be interpreted b y a series of probabilistic (statistical) statements. A deterministi c (sometimes called a mechanistic) model will usually not carry a probabilistic statement. “For mechanistic modeling of common environmental problems, one or more suitable model frameworks may

exist. Many

existing model frameworks in the public domain can be used in environmental assessments. Several institutions, including EPA, develop and maintain these model frameworks on an ongoing basis. Ideally , more than one model framework will meet the project needs, and the project team can select the best model for the specified problem. Sometimes no model frameworks are appropriate for the task, and the team will develop a new model framework or modify an existing framework to include the additional capabilities needed to address the project needs” (EPA 2009b). “During problem specification, the project team defines the regulatory or research objectives, the ty pe and scope of model best suited to meet those objectives, the model performance and data criteria, the model’s domain of applicability , and any programm atic constraints. These considerations provide the basis for developing a conceptual model, which depicts or describes the most important behaviors of the sy stem, object, or process relevant to the problem of interest. Problem specification and the resulting conceptual model define the modeling needs that the project team can then determine whether an existing model can be used to meet those needs or whether a new model should be developed.”(EPA 2009 b ) Of importance to model modification is consideration of the original model origin and purpose. The assumptions used in constructing the original model may be very different than those being used now. The effectiveness of new models built on existing models using different assumptions can lead to erroneous conclusions. For models to be developed, modified, and evaluated, objectives and performance criteria should be defined for both the inputs and outputs of the models. Problem s pecification and identification of m odel p urpose and s cope From Chapter 3.2, E PA 2009b: “Problem specification, culminating in development of the conceptual model, involves an iterative, collaborative effort among model developers, intended users, and decision
- makers to specify all aspects of the problem that will inform subsequent selection or development of a model framework. (Manno et al. 2008).” There are three important planning areas the QAPP needs to cover: objectives, important functions, and model scope.QAPP Handbook 76 December 2012 Agency Review Draft Objectives:

 the purpose or general objectives of the project/model, and the expected applicability or scope of the model and this includes general statements such as what questions the model is designed to answer ; and
 sufficient background information to provide a historical and scientific perspective f or context setting for this particular project , and to act as a basis for future related projects
. Important Functions:  identification of the key functions or outcomes the software must perform, key decisions to be made, actions that need to be taken, and an outline of the outcomes expected from the information ob tained from modeling activities. Model Scope:  the spatial and temporal limits for the model including any key assumptions on model applicability

to the particular and any limitations on model a pplicability

for th is modeling process. Model development, modification, or selection process For model development a clear statement of the basis for the model including its mathematical structure, how software will be used in the development of the new model, sy stem requirements, and procedures for software version control is needed
. Itis essential that the QAPP addresses:

 the theory

that forms the basis for the model, the mathematical algorithms, and approaches used in executing the model computation s;  the existing software to be used in development of a new model, how it will be used , wh y it was selected (subject to the requirements of CIO 2123.0) , and model scope (e.g., spatial, temporal and process detail);  the computer hardware and operating s ys tem requirements for the software application together with the model code development and verification plans, and the most important functions that the software application must perform; and  how the model structure reflects all the relevant inputs to the conceptual model and the expected applicability

and limitations of model use. 4 .2.5.2 QAPP Requirements for Model Application This section expands on the demands of Standard B3.4.4.2, QAPP Req uirements for Model Application and integrates specific advice with the Council for Regulatory Environmental Modeling “Guidance on the Development, Evaluation, and Applicati on of Environmental Models
. ”QAPP Handbook 77 December 2012 Agency Review Draft “The scope (i.e., spatial, temporal and process detail) of models that can be used for a particular application can range from very simple to very

complex depending on the problem specification and data availability , among other factors. When different typ es of models may be appropriate for solving different problems, a graded approach should be used to select or develop models that will provide the scope, rigor, and complexity appropriate to the intended use of and confidence needed in the results.” (EPA 2 009b) Many of the QAPP requirements for model application s are similar to those needed for model identification, and should be noted in the QAPP. The QAPP should document the “how and why” the model will be useful in meeting the project’s objectives. An effective QAPP should be such that a reviewer or interested reader should be able to understand the importance or context of the project. The QAPP has to document the specifications for model application. Problem Selection, Model Purpose, and Model Selection There are two key

areas the QAPP needs to cover: the basis for selection of model, and purpose of model application and anal ys is
. Basis for Model Selection:

 the rationale for choosing this particular model from other competing models; “In all cases, the QAPP should clearl y document for the selected model, why and how the model can and will be used, the spatial extent and resolution, and the temporal extent (length of the modeling period)” (EPA 2009b). Purpose of Model Application:
 the research or re gulatory objective of the application or the purpose for applying the model for this particular project. Model Application For model application the two most important areas the QAPP must define are scenarios, and results
. Scenarios:  it is rare that one model can fit multiple scenarios and so it important that the QAPP accuratel y defines the probable simulation scenarios to which this model will be applied. This documentation will help prevent misunderstandings as to what the model appli es to. Modeling Analysis and Results  the modeling anal ys is and discussion of results is very important to establishing the validity of this particular application. Documentation of these results is an essential part of model development and application.QAPP Handbook 78 December 2012 Agency Review Draft 4.2.6 Data/Project Quality Objectives and Acceptance Criteria In many cases the dataset being considered for use is incomplete in some aspect. Such datasets are often “repaired” using imputation techniques such as nearest neighbor averaging or estimated parameter forecasting. It is essential that the QAPP should document both the extent of imputation and the methodology used. 4.2.6.1 Data Requirements for Model Development or Modification It is acknowledged that when considering existing data for potential use in a modeling project, quantitative acceptance criteria may be difficult to construct. However, some concept of what the acceptable range of acceptance criteria needs to be developed b y the project team. Reliance on broad statements such as “we’ll use the best we can find” or “whatever is in the literature” should be discouraged. If the quantitative criteria are poorl y defined, then an iterative approach may be appropriate where the characteristics of one existing data set are compared to those of ano ther until the “best” is selected. It is advised that the acceptance criteria be phrased in a comparative fashion b y s electing an important DQI and then addressing the MQO for that indicator. For example, “the dataset having the most precise estimate of variance will be selected”, the desired indicator, precision, is the acceptance criterion for the different datasets under consideration. Qualitative acceptance criteria (for example, representativeness, or comparability) may be difficult to describe, but ne ed to be couched in terms that clearl y indicate why one set of existing data may be preferred to another. For example, “the project’s objectives require the cohort of data be exposed to defined environmental conditions for a period not exceeding 5 ye ars”, the selected indicator, comparability , being linked to the objectives of the project. When an existing dataset lacks information on MQOs, careful attention should be given to the overall DQOs of this dataset, because the set of MQOs chosen for the key indicators directl y affect the DQOs. A very

“tight” achievement of the DQOs would be a good augury of the quality of the dataset, a “looser” achievement, less so. While historically

considerable attention has been directed to the principal DQIs of precision, bias, representativeness, comparability , completeness, and sensitivity , it is really representativeness that is probably the single most important indicator of data quality
. Representativeness is a qualitative measure of the degree to which data accuratel y and precisel y represent a characteristic of a population. If the data used in the model fail to be representative of what was intended, no amount of mathematical manipulation or adjustment can make the results valid. For modelers, the DQIs of calibration technique, uncertainty anal ys is, sensitivity anal ys is, and qualitative comparisons to other studies become more important. The QAPP has to set both quantitative and qualitative criteria with what is likely to be available from data anal ys is or available through qualitative comparisons. In either case, acceptance criteria need to be documented (see also section 3.4 of this guidance).QAPP Handbook 79 December 2012 Agency Review Draft 4.2.6.2 Criteria for Acceptance for Model Application The criteria for model application are similar to those encountere d for model development or modification and so the opening remarks of section 4.2.6.1 apply to model application as well. In this element, the QAPP should indicate how “good” the input data and the model need be to answer the question, resolve the problem , or support the decision at the level of quality desired. The intended use of the model should be clearl y s tated and i f used to extrapolate beyond the range of data entered into the model, the appropriate caveats should be included. For each type of data or aspect of the model, indicate how accurate it needs to be and how to ensure the integrity of the information so that it can be relied upon and used for the intended purpose. Without making this clear, data or model outputs may be inadequate for final us e, or conversel y, more expensive to collect or produce than necessary because a higher standard of quality is targeted than is required for the project purposes. The QAPP should clearly state the criteria for acceptance of the model performance or model results. Acceptance criteria are the specific limits, standards, goodness
- of
- fit, or other criteria on which a model will be judged as properl y calibrated (e.g., the percentage difference between reference data values from the field or laboratory and predi cted results from the model). This includes the ty

pes of data and other information that will be necessary to acquire in order to determine that the model is properl y calibrated (e.g., field data, laboratory data, and predictions from other accepted models [see Section 4.3.1]). In addition to addressing these questions when establishing acceptance criteria, the QAPP should document the likely consequences (e.g., incorrect decision making) of selecting data that do not satisfy one or more of these areas (e.g ., are non - representative or are inaccurate), as well as procedures in place to minimize the likelihood of selecting such data. “If the team is evaluating multiple model frameworks, it may be useful to statistically

compare the performance of these compe ting models individually and in aggregate with observational, field, or laboratory

data.” (EPA 2009b) If that is the case, the QAPP should state how those comparisons will be conducted and evaluated. More detailed documentation of data acquisition and use procedures is described in Section 4.3. 4.2.7 Special Training Requirements and Certification Special training or certifications are sometimes necessary for project personnel. For modeling projects, this may

include training in the use of certain models, software, or programming languages; or the necessity for Confidential Business Information security clearance. The QAPP should also specify

how this information will be documented, where the records will be kept, and indicate who is responsible for ensurin g that these special training and certificat e needs are met and documented.QAPP Handbook 80 December 2012 Agency Review Draft 4.2.8 Documentation and Records Requirements The QAPP should describe the process and responsibilities for ensuring that project personnel will receive the most recently approved project documents such as the QAPP, SOPs, and other documents used throughout the project operation. QAPP authors should discuss how these documents will be updated and how information regarding updates will be communicated. The QAPP should indicate where all project documents such as the QAPP or final report and records such as statutory

requests and training records will be stored and for how long. Include backup procedures for any data stored electronically and cite the protocols for access to, retrieval from, and photocop yi ng of information archives. In some cases retention, access to, and final disposition of some records may be regulated. In those cases, this element should address and comply with all relevant regulations. The QAPP should identify th e information to be documented and can include:
 a description of the reporting format for model development/modification, evaluation and application, and for model inputs and model outputs;  identification of other project records to be maintained, how/where the records will be stored, and the length of storage time; ty pes of records may

include: o technical reviews, model tests, data quality assessments of output data and results, along with signatures of individuals performing review and approval, o candidate model assessments for model selection, including references, o actual input used and databases used, o SOPs for model development and evaluation, o pre - and post
- software development information, o spreadsheet data files containing monitoring data, o copies of model ing reports, and o model code, equations, and user’s guide (including information on the location of earlier versions)
. The QAPP should indicate where all project documents (e.g., QAPP or final report) and records (e.g., statutory

requests or training record s) will be stored and for how long. The QAPP should identify any requirements set by

statute or policy together with information on access, retention, and final disposition. **4.3 DATA ACQUISITION: MO**  **DEL DEVELOPMENT, MOD IFICATION, AND USE**  **(D O**  **)** For modeling p rojects there is less of a clear line between “plan” and “do” than for projects involving simply data. This section offers advice on the Standards Clause 2.1, Scope, Standards Clause 7.7, Systematic Plan ning , Annex B3.4, Data Acquisition, and Annex B4.4, Development and Use of Mathematical and Electronic Models. Some of the elements listed in the “plan” part could be shifted to the “do” part depending on the intent of the modeling project. For example, some of Data Requirements for Model Development or Mo dification (4.2.6.1) or Criteria for Acceptance for Model Application (4.2.6.2) could be placed here or even in the “check” section (4.4). The exact placement depends on the objectives and logistics of model development.QAPP Handbook 81 December 2012 Agency Review Draft The elements in this section addres s all aspects of data production and collection to ensure that appropriate methods for sampling, measurement and anal ys is, data production, data handling, and QC activities are employed and documented. The following QAPP elements describe the actual method s or methodology to be used for the collection, handling, and anal ys es of samples, and the management (i.e., compiling and handling) of the data. The selection of data to be used in any modeling project is critical to meaningful interpretation of the results of the project. Knowing who originated (collected or was responsible for collecting) the proposed data can be very informative. Data originators might be EPA, states, Tribes, municipalities, potentially responsible parties (PRPs), or activist group s, for example. Confidence in the data may , to some small extent, be judged according to the data originator. That is not to say

that data collected b y states, for example, should always be accepted, or that data collected b y PRPs, for example, should always be excluded. However, knowing who the data originator was may

offer some information as to the purpose of the data collection and reflect on the possible impartiality of the lead author or source of the data. If the data to be used comes from an EPA so urce, then the method of data collection, the results, and limitations on use of the results should be available in the data’s QAPP. Reference to the QAPP and its location should be made, and important extracts from that data QAPP included in the modeling QAPP. If the data source is outside EPA attention should be given to the issues raised in Chapter 3.3 of this Handbook (Data Acquisition, Existing Data). If new data are to be generated for use in modeling, a separate QAPP for the data will probabl y be n eeded. If existing data are to be used, the information in Chapter 3, Section 3.4 and 3.5 should be incorporated into the modeling QAPP. The methods described in these elements should have been summarized earlier in Section 4.2.6. The purpose here is to p rovide detailed information on the methods. If the designated methods are well documented and are readil y available to all project participants, citations are adequate; otherwise, detailed copies of the methods or SOPs should accompany the QAPP either in the text or as attachments. 4.3.1 Data Requirements for Model Input For projects involving modeling, the data to be used in the model should meet specified objectives, as should the outputs of the model. The QAPP should define how the data quality acceptance criteria will be determined (for all information to be collected including information obtained from previous studies). The QAPP should also explain how the data acceptance criteria relate to the desired quality of model outputs. 4.3.1.1 QAPP Re quirements for Model Development or Modification F or model development, Annex B3.4.4.1 indicates the QAPP must define:
 the data needs/inputs/sources;  any boundary

or initial condition specifications;  data reporting requirements and format for model inputs/ outputs; and  the data quality objectives for model input and data used for model development.QAPP Handbook 82 December 2012 Agency Review Draft The QAPP should document the quality and quantity

of data necessary

for model development, modification, use, or evaluation. DQOs and specified acceptable range o f uncertainty for the model input data should be defined as they “provide the specifications for model quality and associated checks. Well
- defined DQOs guide the design of monitoring plans and the model development process (e.g., calibration and verificati on). The DQOs provide guidance on how to state data needs when limiting decision errors (false positives or false negatives) relative to a given decision. The DQOs should include a statement about the acceptable level of total uncertainty that will still enable model results to be used for the intended purpose. Uncertainty describes the lack of knowledge about models, parameters, constants, data, and beliefs. Defining the ranges of acceptable uncertainty  either qualitatively or quantitatively

 helps proj ect planners generate “specifications” for QA planning and partiall y determines the appropriate boundary or initial condition specifications and complexity for [model development].” (EPA 2009 b ) For developing or modifying models, any boundary or initial co ndition specifications should be defined. In applying a model or software, availability of data and knowledge of inputs needed is central. Data quality and quantity should be noted, along with procedures for excluding data. 4.3.1.2 QAPP Requirements for Mo del Application F or model application, Annex B3.4.4.2 gives the data requirements by focus ing on data reporting requirements , format for model inputs and outputs , and discussion of issues regarding information on how data will be acquired and used in the project:
 the need and intended use for each type of data or information to be acquired;  selected source of the data and how it will be differentiated from other sources in the data management s ys tem;  how the data will be identified or acquired together with the basis for exclusion of competing datasets ;  the method of determining the underl yi ng quality and quantity of the data; and  the criteria established for determining whether the level of quality for a given set of data is acceptable for use on the project. 4.3.1.3 DQIs for all Ty

pes of Modeling Projects Some of the k ey

criteria for individual data values, or a set of data values, i nclude an examination of their D QIs with respect to potential use in modeling projects : Precision: Is the estimate of variability small enough to meet the uncertainty objectives of the modeling project? Bias: Would any

characteristics of the dataset dire ctly impact the model output (e.g., unduly high or low process rates)? For example, was bias in result analy sis documented? I s there sufficient information to estimate and correct bias? If using data to develop probabilistic distributions, are there adequate data in the upper and lower extremes of the tails to allow for unbiased probabilistic estimates?QAPP Handbook 83 December 2012 Agency Review Draft Representativeness: Were the data collected from a population sufficientl y similar to the population of interest and the model
- specified population boundaries? Were the sampling and anal ytical methods used to generate the collected data acceptable to this project? Are there potentially

confounding effects in the data (e.g., season, time of day , location, and scale incompatibilities) that need to be addressed so as to not unduly impact the model output? Comparability: Are there concerns about comparing the data used in modeling to current conditions with respect to age or changes in modeling practices? Are there possible concerns that the structure of the population from where the data was taken could be different from current conditions? Completeness: Are there any gaps in the data record that could cause concerns with the modeling output? To what extent was data interpolation or imputation necessary before th e data were suitable for modeling purposes? Sensitivity: Have the data been evaluated in a manner that permits perturbation analy sis or estimates made as to the sensitivity of the model output to parameter changes? I s the s ys tem of qualifying or flagging d ata adequately documented to allow data from different sources to be used on the same project (e.g., distinguish actual measurements from estimated values and note differences in detection limits)? The influence and relative importance of the DQIs depends on the purpose and nature of the modeling project and further information may be found in Appendix B of this guidan ce. When input data are derived from existing models care has to be taken to ensure evaluation of all stages of model development: “Some asse ssments require linking multiple model frameworks such that the output from one model is used as input data to another model. For example, air quality modeling often links meteorological, emissions, and air chemistry /transport models. When employing linked models, the project team should evaluate each component model, as well as the full sy stem of integrated models, at each stage of the model development and evaluation process” (EPA 2009b)
. The importance of knowing the nature and characteristics of the input data is crucial to the interpretation of the project’s results. 4.3.2 Data Management Tasks The QAPP should discuss how data will be managed, how the path of data production in the field or laboratory will be traced for the entire project . This includes the following:
 describe or reference the standard record
- keeping procedures, and discuss the approach to be used for data storage and retrieval of electronic media (see Brilis, et al, 2004) ;  discuss the plan for detecting and correcting errors from data c onversion (e.g. metric/English, units
- to - units, and significant figures) as well as for preventing loss of data during reduction, reporting, and entry to forms, reports, databases;  discuss the plan for detecting and correcting errors from data migration to new hardware/software (including checks to verify integrity and non
- corruption);QAPP Handbook 84 December 2012 Agency Review Draft  describe all data
- handling equipment and procedures to process, compile, analyze, and interpret the model data, including any necessary computer hardware and software;  addre ss any specific data
- management performance requirements and describe the procedures that will be followed to demonstrate acceptability of the necessary hardware/software configuration; and  identify

who is responsible for each data
- management task together with frequency for performing these tasks
. In addition to describing the data management tasks, the QAPP should also address how these tasks will be documented throughout the project, not already addressed in Section 4.2.8.
**4.4 A**  **SSESSMENTS: M ODEL E**  **VALUATION ACTIONS**  **(C HECK**  **)** The third part of the Plan
-Do
- Check
- Act paradigm assesses the model by considering the output, especially with respect to deviations from what was anticipated
. The advice given here is drawn from the Standards Clause 7.8, Assessment of Data and Information, and the assessment parts of Annex B3. 4.4.1, QAPP Requirements for Model Development/Modification, and B3.4.4.2 QAPP Requirements for Model Application
. The elements in this section address project checks to see if the model will conform to the project’s objectives, and to estimate the effect of any deviations. These activities may be performed throughout the project to inform mid
- course corrections o r at the completion of the project. The roles and responsibilities o f project personnel involved in assessing modeling activities should be defined in this section. This should include the procedures for corrective actions and to whom reports are made. To the extent that the modeling activity includes acquiring data or the use of existing data, the assessment methods outlined in the previous two chapters are appropriate. This section is guidance for the evaluation of the modeling activity
. 4.4.1 Model Evaluation Methods and Activities Model evaluation is defined as the pro cess used to generate information to determine whether a model and its analy

tical results are of a quality sufficient to serve as the basis for a decision. Model evaluation is conducted over the life cycle of the project, from development through applicati on using the performance criteria established
. The effectiveness of evaluation depend on good documentation of both quantitative and qualitative comparisons to the performance criteria. Model evaluation provides information to help answer four main questions (Beck 200 0 ) : 1. How have the principles of documented science been addressed during model development? 2. How is the choice of model supported b y the quantity and quality of available data? 3. How closel y does the model approximate the real system of intere st? andQAPP Handbook 85 December 2012 Agency Review Draft 4. How does the model perform the specified task while meeting the objectives set by QA project planning? These four factors address two aspects of model quality . The first factor focuses on the intrinsic mechanisms and generic properties of a model , regardless of the particular task to which it is applied . In contrast, the latter three factors are evaluated in the context of the use of a model within a specific set of conditions . Hence, it follows that model quality is an attribute that is meaningful only

within the context of a specific model application . A model's quality to support a decision becomes known when information is available to assess these factors. Evaluation of a regulatory

model builds upon the four factors described previously and should continue throughout the life of the model. As recommended b y the National Research Council (NRC 2007), the QAPP should contain a detailed model evaluation plan that includes (but is not limited to):
 a description of the model and its intended uses;  a description of the model’s relationship to data, including the data for inputs, calibration, and corroboration;  a description of how such data and other sources of information will be used to assess the ability of the model to meet its intended task;  a description of all the evaluation plan elements by using an outline or diagram that shows how the elements relate to the model’s life cycle;  a description of the factors or events th at might trigger the need for major model revisions or the circumstances that might prompt users to seek an alternative model (these can be fairl y broad and qualitative); and  identification of the responsibilities, accountabilities, and resources needed to ensure implementation of the evaluation plan. The goal of model evaluation is to ensure model quality . Documentation of the plan for model evaluation provides assurance that the desired model quality will be achieved. 4.4.2 Evaluation of the Model 4.4.2.1 Evaluation of Model Development or Modification Standard Clause B3.4.4.1 states that the evaluation of a model should focus on documented science, testing strategies and procedures, and documentation of results. The QAPP should describe how the pro

ject team will ensure that the following issues will be evaluated and documented during model development or modification, as appropriate:
 documentation of the science (including peer review of the theory and equations) underlying the model and appropriateness of model complexity for the problem;  testing strategies including individual module tests, integration tests, sy stems testing, acceptance testing, and beta testing, as applicable; data quality and quantity objectives to support the model choice;QAPP Handbook 86 December 2012 Agency Review Draft  the p rocedure for each test shall be provided and the process of confirming the test results included (i.e., evaluation criteria are to be identified during the initial stages of model development);  the use of sensitivity

and uncertainty analysis with consistency of model structure with all relevant inputs described in the conceptual model;  implementation process for the software and any applicable development standards with identification of the model code and code verification, if completed;  internal quality checks applied during the development process (e.g., design and code verification, configuration control procedures, and change control procedures);  procedures for controlling, documenting, and archiving all significant changes to the software and hardware ; and  requirements for project documentation (e.g., design document, source code, programmer’s manual, or user guide). 4.4.2.2 Evaluation of Model Application It is important to note the iterative nature of modeling: “Once a model has been accepted for use by decision
- makers, it is applied to the problem that was identified in the first stages of the modeling process. Model application commonly involves a shift from the hindcasting (testing the model against past observed conditions) used in the model devel opment and evaluation phases to forecasting (predicting a future change) in the application phase. This may involve a collaborative effort among modelers and program staff to devise management scenarios that represent different regulatory alternatives (EPA 2009b)
. The needs (Standards Clause B3.4.4.2 ) of a QAPP center on:
 the quality

objectives to be used in the mod el evaluation;  the requirements for qualitative and quantitative model corroboration;  use of sensitivity

and uncertainty

anal ys is;  requirements for model post
- auditing;  how the model evaluation will be documented;  the model parameters and method of estimation; and  the model calibration methods. S ome model applications may entail trial
- and
- error model simulations, where model inputs are changed it erativel y until a desired or stable condition is achieved. When a sufficient quantity of data is available , it is common practice to use a portion of their existing data to establish the model, and then use the remaining portion to evaluate the performance of the model. The rationale on what proportion of data is used to establish the model and how that data is selected (randoml y or serially across time for example) should be documented. When the model generates new data based on the use of existing data, the data fields (variables) and algorithms should be traceable back to their origin for this helps prevent the misinterpretation of the project results
. The importance of documentation can be seen in EPA 2009b: “Using a model in a proposed decision requires that the model application be transparentl y incorporated into the public process.QAPP Handbook 87 December 2012 Agency Review Draft This is accomplished by

providing written documentation, in the QAPP, of the model’s relevant characteristics in a styl e and fo rmat accessible to the interested public, and by sharing specific model files and data with external parties, such as technical consultants and university scientists, upon request .” Finally , it is important to note how the data used to evaluate the model compares to the actual scenario the model will be applied to. A model that has good evaluation properties for statistics such as averages may

be quite poor for predicting extremes. The acceptance criteria established should be directl y related to the eventual use of the model. 4.4.3 Model Output Sensitivity and Uncertainty

Anal ys is Sensitivity

analy sis (Standard Clause B3.4.4.1) is the study of how a model’s response can be attributed to known changes in model inputs. Sensitivity anal ys is is recommended as the principal evaluation tool for identifying the most and least important sources of uncertainty in environmental models. From EPA 2009 b : “Uncertainty anal ys is investigates the lack of knowledge about a certain population or the real value of model parameters. Uncertainty can sometimes be reduced through further study and b y collecting additional data. EPA guidance (e.g., EPA 1 997 b ) distinguishes uncertainty analysis from methods used to account for variability in input data and model parameters. [V]ariability

in model parameters and input data can be better characterized through further study but is usually not reducible.” (EPA 2009 b ) “Although sensitivity

and uncertainty analyses are closel y related, sensitivity is algorithm
specific with respect to model “variables” and uncertainty is parameter
- specific. Sensitivity anal ys is assesses the “sensitivity” of the model to specific parameters and uncertainty analysis assesses the “uncertainty” associated with parameter values. Both ty pes of anal ys es are important to understand the degree of confidence a user can place in the model results” (EPA 2009 b )”. Advice on conducting uncertainty analysis and the establishment of acceptable levels of total error (uncertainty) may

be found in EPA 2009 b , and a dditional information on uncertainty anal ys is in Good Modelling Practice Handbook (DDPW 1999), and Uncertainty Anal ys is Guidance (EU 2007)
. **4.5 REVIEW, EVALUATION O**  **F USABILITY : MODELING RESULTS U**  **SABILITY AND REPORTING REQUIR** **EMENTS (A**  **CT )**  The final part of the paradigm concerns the reconciliation of the model outputs to the overall project objectives. Peer review is an important part of modeling and must be comprehensively addressed. This section offers guidance on Standards Clause 7.9, the evaluation parts of Annex B3.4.4.1, QAPP Requirements for Model Development/Modification, B3.4.4.2, QAPP Requirements for Model Application, and B3.6, Review, Evaluation of Usability, and Reporting Requirements. The QAPP should state how the results obtained from the project will be reconciled with the project objectives and acceptance criteria. This includes the proposed methods for anal yz ing the modeling results and for determining possible anomalies or limitati ons on the use for the intended purpose.QAPP Handbook 88 December 2012 Agency Review Draft The QAPP should describe how departures from assumptions established in the planning phase of the modeling process will be assessed, how anomalies will be resolved, and how limitations on the use of the data from anomalies and departure from assumptions will be reported to decision makers. 4.5.1 Peer Review For all models, p eer r eview (Standard Clause B3.4.4.2) is an important aspect for acceptance of both model development/modification and application as it acts as an independent form of QA to ensure the validity

of results. “Peer review provides the main mechanism for independent evaluation and review of environmental models. Its purpose is two
- fold:
 to evaluate whether the assumptions, methods, and conclusions derived from environmental models are based on documented scientific principles; and  to check the scientific appropriateness of a model for informing a specific regulatory decision (the latter objective is particularl y important for secondary applications of existing models). In formation from peer reviews is also helpful for choosing among multiple competing models for a specific regulatory

application. Finall y, peer review is useful to identify the limitations of existing models. Peer review is not a mechan ism to comment on the regulatory decisions or policies that are informed by models (EPA 200 6 c). Peer review charge questions and corresponding records for peer reviewers to answer those questions should be documented during the planning stage. For example, peer reviews may focus on whether a model meets the objectives or specifications that were established for the project. All models that inform significant regulatory decisions are candidates for peer review for several reasons:
 model results will be used as a basis for regulatory or policy /guidance decision making;  these decisions likely

involve significant investment of Agency resources; and  these decisions may

have inter
- Agency

or cross - agency

implications/applicability
. It is recommended that “a new model should be scientificall y peer - reviewed prior to its first application; for subsequent applications, the program manager should consider the scientific/technical complexity and/or the novelty

of the particular circumstances to determine whether additi onal peer review is needed (EPA 1993).” (EPA 2009 b ) “Models used for secondary applications (existing EPA models or proprietary models) will generall y undergo a different type of evaluation than those developed with a specific regulatory information need i n mind. Specificall y, these reviews may deal more with uncertainty about the appropriate application of a model to a specific set of conditions than with the science underl yi ng the model framework.” (EPA 2009 b )QAPP Handbook 89 December 2012 Agency Review Draft The following aspects of a model should be peer - reviewed to establish scientific credibility (EPA 1993):
 appropriateness of input data;  appropriateness of boundary condition specifications;  documentation of inputs and assumptions;  applicability

and appropriateness of selected parameter values;  e valuation of model uncertainty ;  documentation and justification for adjusting model inputs to improve model performance (calibration);  model application with respect to the range of its validity ; and  supporting empirical data that strengthen or contradic t the conclusions that are based on model results. To be most effective and maximize its value, external peer review should begin as early in the model development phase as possible. The importance of using independent model review is extremely important and the decision whether to utilize the Agency ’s Peer Review sy stem made as soon as possible (EPA 2006c). 4.5.2 Description of Model Documentation The importance of documentation can be seen from EPA 2009 b : “Documentation enables decision makers and other model users to understand the process b y which a model was developed and used. During model development and use, many choices are made and options selected that may

bias the model results. Documenting this process and its limitations and uncertainties is e ssential to increase the utility and acceptability

of the model outcomes. Modelers and project teams should document all relevant information about the model to the extent practicable, particularly when a controversial decision is involved. In legal procee dings, the quality

and thoroughness of the model’s written documentation and the Agency ’s responses to peer review and public comments on the model can affect the outcome of the legal challenge.” The QAPP should describe how the modeling process, the mode l framework, the model, the sensitivity

and uncertainty

analyses, and the model outputs will be documented. The documentation should include a clear explanation of the model’s relationship to the scenario of the particular application. This explanation sho uld describe the limitations of the available information when applied to other scenarios. The evaluation, verification, and validation performed on the model should be documented. This documentation should include a clear explanation of the model’s relati onship to the final goal of the project or task. Disclosure about the state of science used in a model and future plans to update the model can help establish a record of reasoned, evidence
- based application to inform decisions. It is also advisable to in clude in the QAPP any particular memory

requirements, necessity for specific software for model application, and any i ssues with model portability
.QAPP Handbook 90 December 2012 Agency Review Draft 4.5.3 Specifications for Model Maintenance and User Support The project team should determine and document in the QAPP, during the planning stage, the expected maintenance and user support needed throughout the life cycle of the model. 4.5.4 Reports to Management The QAPP should identify the frequency

and distribution of reports issued to inform management (EPA or otherwise) of the project status. QAPP authors may choose to identify

the preparer and the recipients of the reports, and any specific actions recipients are expected to take as a result of the reports. Periodic QA management reports are intended to e nsure that managers and stakeholders are kept informed of project status and results of all QA assessments. Efficient communication of project status and problems will allow project managers to implement timely and effective corrective actions so that data produced for the project can meet the DQOs. The QAPP should describe the content of each QA management report that will be generated for the project, including an evaluation of measurement error as determined from the assessments. Assessment checklists, reports, requests for corrective action letters, and the corrective response letters should be included as attachments or referenced in the QA management reports. The following issues may be included in QA management reports:
 Graphs, charts, diagrams helpin g interpreting results;  conformance of project activities to QAPP requirements and procedures;  status of project and schedule delay s;  deviations from the approved QAPP and approved amendments to the QAPP;  results of interim model evaluation activities and corrective actions;  model sensitivity

and uncertainty analyses; and  limitations on the use of model output. Although these issues listed may be addressed in QA management reports, they should also be included in the QA/QC section of the final project report. The final project report should, at a minimum, give a reconciliation of project data with project objectives, data summary (including tables, charts, and graphs), a summary of major problems encountered and their resolution, and any additional data quality

concerns together with conclusions and recommendations
.QAPP Handbook 91 December 2012 Agency Review Draft **GLOSSARY** Several definitions given in CI O Policy 2106.0, CIO Procedure 2106
- P - 01.0, CI O Standard 2106 - S - 01, and CIO Standard 2106
- S - 02 are repeated below for added clarity and utility

to this Guidance. **Approved:** The documented determination that the proposed qual ity document is suitable for the intended purpose and meets the requirements is specified in the applicable Quality Standard. **Assessmen t:**  The evaluation process used to measure the performance or effectiveness of a sy stem and its elements. As used here, a ssessment is an all
- inclusive term used to denote any of the following: audit, performance evaluation, management review, peer review, inspection, or surveillance. **Conceptual M**  **odel** : This describes the scientific and engineering process under investigation . It is used as a tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. **Corrective Action :**  An action to eliminate the cause of a nonconformity or undesirable situation and to prevent recurrence. **Data Standard** : A documented consensus
- based agreement on the format and definition of data. **Data Usability:** The process of determining and ensuring that the quality of the data produced meets the intended use of the data. **Document:** Recorded information regardless of phy sical form or characteristics including individual records or items of non
- record materials. **Environmental Data :**  Any data or information pertaining to the environment that describe measured outputs from processes; environmental conditions in a specific location; ecological effects and consequences; health effects and consequences; biological, chemical, and radiologi cal conditions; or the performance of environmental technology
. E nvironmental data include information collected directl y from measurements, produced from models, and compiled from other sources such as databases, informati on sy stems, literature, or the i nternet. **Environmental Data Collection:** The process of acquiring or gathering environmental data through various means including, but not limited to, sampling and anal ys is activities, retrieval from information sy stems and literature sources, and receipt f rom EPA partners and the regulated community
. **Environmental Data Operations:** The work performed to collect, produce, or use environmental data.QAPP Handbook 92 December 2012 Agency Review Draft **Environmental Data Production:** The process of generating environmental data through various means including, but not limited to, the use of measurement instrumentation, information technology , computer models, and data anal ys is tools (e.g., statistics, risk assessment methods). **Equiva lent Document:**  A set of documents that contains all the information and management controls (signatures) as the required documents used in the Standard. **Extramural Agreement:** Legal agreement between EPA and a non
- EPA organization for the acquisition of it ems or services b y EPA or financial assistance to a non
- EPA organization. Such agreements include acquisition agreements (e.g., contracts, work assignments, delivery orders, task orders), assistance agreements (e.g., cooperative agreements, research grants , state and local grants), and EPA
- funded IAs with other governmental entities. **Graded Approach:** The process of basing the level of application of managerial controls applied to an item or work according to the intended use of the results and the degree o f confidence needed in the quality of the results. **Guidance:** A non - mandatory compilation of advice, examples, best practices, or past experience. Guidance may

supplement procedures. **Handbook:** A non - mandatory compilation of advice, examples, best practice s, or past experience s, may

be revised according to the issuing Office’s Peer Review Policy
. **Information:** For purposes of this Standard, information means any communication or representation of knowledge such as facts or data, in any medium or form, including, but not limited to, textual, numerical, graphic, cartographic, narrative, or audiovisual forms. **Inf ormation System**  : An organized collection, storage, and presentation sy stem of data for decision making, progress reporting, and for planning and evaluation of programs. It can be either manual or computerized, or a combination of both. **Information Technol ogy:**  The study, design, development, implementation, support, or management of computer
- based information sy stems, particularl y software applications and computer hardware. **Informative:** Non - mandatory advice on how to achieve a goal or meet a requirement. **Integrity (Information):** The assurance that the information is protected from unauthorized access or change and is not compromised through corruption or falsification. **Life Cycle:** The life span of a product or service from its initial planning and develop ment, to its use and maintenance, and to its final closure or disposal. **Management Controls:** A sy stem of management functions to enable managers to determine that the operations of a program or organization satisfy predetermined goals and objectives, that performance is in line with standards and specifications, and to implement any remedial actionsQAPP Handbook 93 December 2012 Agency Review Draft needed to ensure that human and other resources are being used in the most effective and efficient way

possible in achieving the organization’s mission. **Manag ement System**  **:** A sy stem to establish policy and objectives and to achieve those objectives (ISO 9001). A management s ys tem may describe the policies, objectives, principles, authority , responsibilities, accountability, and implementation plan of an organization for conducting work and producing products and services. Management sy stems include I SO 9001 on quality management, ISO 14001 on environmental management, and OHSAS 18000 on occupational health and safety. **Metadata:** The information about data required to facilitate its use, understanding, and management. For example, metadata should answer questions about data such as why the y were collected, how they

were collected and by

whom, what was done to the data, what they were used for, what were thei r limitations, and what were the acceptance criteria. **Model:** A simplification of reality that is constructed to gain insights into select attributes of a phy sical, biological, economic, or social sy stem. A formal representation of the behavior of sy stem p rocesses, often in mathematical or statistical terms. The basis can also be physical or conceptual. **Normative:** A mandatory set of rules that must be followed. **Organization:** A company , corporation, firm, enterprise, or institution, or part thereof, whethe r incorporated or not, public or private, that has its own functions and administration. In the context of this Standard, an EPA organization may be an Office, Region, National Research Center or Laboratory , or a subunit such as a division, branch, section , or team. **Objectivity (Information):** The assurance that information is presented in an accurate, clear, complete, and unbiased manner, and, as a matter of substance, is accurate, reliable, and unbiased. **Peer Review:** A review conducted b y qualified indiv iduals or organizations who are independent of those who performed the work, but are collectively equivalent in technical expertise to those who performed the original work. **Policy:** A high
- level statement about an Agency requirement designed to influence and determine decisions, actions, and other matters. It is usually driven b y statute, executive order, the mandate of an oversight agency or Congress, or the head of the organization. **Practice:**  an expected procedure or way of doing of something.  the carryi ng out or exercise of an expected procedure or way of doing of something. **Procedure:** The required steps, course of action, or processes needed to accomplish or satisfy a policy
.QAPP Handbook 94 December 2012 Agency Review Draft **Process:** A set of interrelated resources and activities which transforms i nputs into outputs. Examples of processes include anal ys is, design, data collection, operation, fabrication, and calculation. **Product:** The intended result or final output of an activity or process that is disseminated or distributed among EPA organizations or outside of EPA. **Quality:** The totality

of features and characteristics of a product or service that bear on its ability to meet the stated or implied needs and expectations of the user. **Quality Assurance (QA):** A management or oversight function that deals with setting policy and running an administrative sy stem of management controls that cover planning, implementation, review and maintenance to ensure products and services are meeting their intended use. **Quality Assurance Manager (QAM) :**  The individual designated as the principal manager within the organization having management oversight and responsibilities for planning, documenting, coordinating, and assessing the effectiveness of the QMS for the organization. N ote: Other personnel having QA or QC duties may be referred to as QA Officer and QA Coordinator. **Quality Assurance Project Plan (QAPP):** A document describing in compreh ensive detail the necessary QA, QC, and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance objectives and criteria. **Quality Control (QC):** The overall sy stem of technical activities that measure the attributes and performance of a process, item, or service against defined standards to verify that they meet the stated requirements established by the customer; operational techniques and activities that are used to fulfill requirements for quality
. **Quality Management:** That aspect of the overall quality management s ys tem of the organ ization that determines and implements the quality policy . Quality management typicall y includes strategic planning, allocation of resources, and other s ys tematic activities (e.g., planning, implementation, and assessment) pertaining to the application of quality practices to the organization’s programs. **Quality Management Plan (QMP):** A formal document or manual that describes the QMS in terms of the organizational structure, functional responsibilities of management and staff, lines of authority , and required interfaces for those planning, implementing, and assessing all activities conducted. **Quality Management System (QMS):** A structured and documented management s ys tem describing the policies, objectives, principles, organizational authority , responsibil ities, accountability , and implementation plan of an organization for ensuring quality in its work processes, products (items), and services. The QMS provides the framework for planning,QAPP Handbook 95 December 2012 Agency Review Draft implementing, documenting, and assessing work performed b y the organi zation and for carrying out required QA and QC activities. **Quality Program :**  The totality of management controls, processes, and documentation in planning, implementation, and assessment of applying quality to the creation of Agency products and services. **Record:** A document stating results retrieved or providing evidence of activities (I SO 9000:2005). NOTE: A federal record is an information resource in any format that is needed to describe Agency

activities (44 U.S.C. § 3301). **Required Element:** An element whose presence is obligatory in a Standard. **Requirement:** An expression of the content of a Standard conveying a criterion to be fulfilled if compliance is to be claimed and from which no deviation is permitted. **Service:** A discrete function that perform s operations and returns a set of results to an external requester. **Specification:** A document stating requirements and which refers to or includes drawings or other relevant documents. Specifications should indicate the means and the criteria for determining conformance. **Standard:** An accepted, consensus
- based specification which defines s ys tems, processes, methodologies, or practices. It provides a basis for assuring consistent and acceptable minimum levels of quality , performance, safety, and reliability. Standards usually are included in or accompany procedures. **Standard Operat ing Procedure (SOP):**  A written document that details the method for an operation, anal ys is, or action with thoroughly prescribed techniques and steps, and that is officiall y approved as the method for performing certain routine or repetitive tasks. **Usabil ity Assessment:**  The evaluation of data based upon the results of data validation and verification for the decision(s) being made. Reviewers assess whether the process execution and resulting data meet quality objectives based on the criteria given in the Q APP. **User:** An organization, group, or individual that utilizes the results from environmental programs , or the customer for whom the results or products were collected or created. **Utility (Information):** The assurance that information is useful for its intended purpose. **Validation (Information):** The confirmation by examination and provision of objective evidence that the particular requirement for which the information is intended are fulfilled; the process of determining whether the specifications were a ppropriate and that the verified results will meet the data user’s needs.QAPP Handbook 96 December 2012 Agency Review Draft **Verification (Information):** The confirmation by examination and provision of objective evidence that validated information fulfills specified requirements; the process of checking whether the information met the project’s specifications as part of an agreement or contract.QAPP Handbook 97 December 2012 Agency Review Draft **REFERE NCES**  **40 CFR 3 ,**  Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Part 3: Special rules governing certain information (from regulatory programs). Washington, D.C. http://ecfr.gpoaccess.gov/ **40 CFR 51**  **Appendix W** , Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Part 51: R equirements f or Preparation , A doption , and S ubmittal of I mplementation Plans. Washington, D
. C. http://ecfr.gpoaccess.gov/ **40 CFR 403.12**  **( b**  **)** , Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Sub chap ter N : Effluent Guidelines and Standards
. Part 403: General Pretreatment Regulations for Existing and New Sources of Pollution. Washington, D
. C. http://ecfr.gpoaccess.gov/ **40 CFR 763** , Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Part 763, Subpart E: Asbestos
- Containing Material in Schools. Washington, D
. C
. http://ecfr.gpoaccess.gov/ **4 9**  **CFR 171,**  Code of Federal Regulations (annual edition). Title 49: Transportation. Part 171: Definition of Hazardous Material. Washington, D.C. http://ecfr.gpoaccess.gov/ **ANSI 2004 ,**  American National Standards Institute and American Society for Quality, Quality System s for Environmental Data and Technology Programs : Requirements with Guidance for Use ( ANSI/ASQ E4
- 2004

). http://asq.org/quality
-press/display **ASTM 1999** , ASTM Standard D4840 - 99 , 1999, Standard Guide for Sampling Chain
- of
- Custody Procedures , ASTM International, West Conshohocken, PA. http://astm.org **Beck, B, 2002**  Model Evaluation and Performance. In: Encyclopedia of Environme trics . A.H. El
Shaarawi and W.W. Piegorsch, eds., John Wiley & Son, New York. **Box, G. E. P., W. G. Hunter, and J. S. Hunter** , 1978. Statistics for Experimenters . John Wiley
. New York. **Brilis, G .**  **M, J .**  **C. Worthington, A. D**  **. Wait, 2000.**  Quality Science in the Courtroom: U.S. EPA Data Quality and Peer Review Policies and Procedures c ompared to the Daubert Factors, Environmental Forensics , **vol. 1** p197 - 203. **Brilis, G .**  **M .,**  **J. Lyon, J.C. Worthington, and R. Lysakowski, 2004.** Auditing Electron ic Record
- keeping Sy stems used in Scientific Research and Development, Quality Assurance , **vol. 11**  p5 - 24 . **Cochran, W. G., and D. R. Cox** , 1957. Experimental Designs
. John Wiley
. New York.QAPP Handbook 98 December 2012 Agency Review Draft **Currie, L. A.** , 1995. Nomenclature in Evaluation of Anal ytical Methods Including Detection and Quantification Capabilities. Pure and Applied Chemistry , **Vol. 67, No. 10** , pp. 1699

– 1723. **DDPW 1999 ,**  Good Modelling Practice Handbook, Dutch Department of Public Works. http://harmoniqua.wau.nl/public/Reports/Existing%20Guidelines/GMP111.pdf **EPA 1997a,** Procedures for Preparing Emission Factor Documents (EPA
- 454/R
- 95 - 015) U.S. Environmental Protection Agency
. http://www.epa.gov/ttnchie1/efdocs/procedur.pdf **EPA 1997 b**  **,** Multi
- Agency Radiation Survey

and Site Investigation Manual (MARSSIM)
. NUREG - 1575, EPA/402/R
- 97/016, U.S. Environmental Protection Agency . www. marssim .com **E PA 1999**  , Contract Laboratory Program, National Functional Guidelines for Organic Data Review. U.S. Environmental Protection Agency
. www.epa.gov/superfund/programs/clp/guidance .htm **EPA 2000 ,**  Guidance on Technical Audits and Related Assessments for Environmental Data Operations , EPA QA/G
- 7 , January 2000 , U.S. Environm ental Protection Agency
. http://www.epa.gov/quality **EPA 2001 a**  **,** Requirements for Quality Assurance Project Plans , EPA QA/R - 5. EPA/240/B 0 1 /00 3 , U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2001b,** ORD Policies and Procedures Manual, Chapter 13 Quality Assurance, U. S. Environmental Protection Agency
. http://dcordhqapps1.eoa.gov/orma/policies.nsf/webpolicy?OpenView **EPA 2002a,** Guidance for Quality Assurance Project Plans , EPA QA/G
- 5. EPA/240/R
- 02/009 , U. S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2002 b**  **,** Guidance for Choosing a Sampling Design for Environmental Data Collection , EPA QA/G
- 5S , U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2002 c**  **,** Guidance for Quality Assurance Project Plans for Modeling , EPA QA/G
- 5M U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2005 a**  **,** Uniform Federal Policy for Quality

Assurance Project Plans , Intergovernmental Data Quality Task Force ( EPA - 505 - B - 04 - 900A ) U.S. Environmental Protection Agency
. http://www.epa.gov/fedfac/documents/qualityassurance.htm **EPA 2005b ,**  Workbook for Uniform Feder al Policy for Quality Assurance Project Plans , Intergovernmental Data Quality Task Force ( EPA - 505 - B - 04 - 900 C) , U.S. Environmental Protection Agency
. http://www.epa.gov/fedfac/documents/qualityassurance.htmQAPP Handbook 99 December 2012 Agency Review Draft **EPA 2005c,** OSWER Directive 9272.0
- 17 Implementation of the Uniform Federal Policy for Quality Assurance Project Plans (UFP
- QAPP) at Federal Facility Hazardous W aste Sites , June 7, 2005, U.S. Environm ental Protection Agency
. www.epa.gov/fedfac/pdf/ oswer _qapp_directive.pdf **EPA 2005d,** OSWER Directive 9272.0
- 20 Applicability of the Uniform Federal Policy for Quality Assurance Project Plans ( EPA 505
- 04
- 900A, December 21, 2005, U.S. Environmental Protection Agency
. www.epa.gov/fedfac/pdf/ oswer _qapp_directive.pdf **EPA 2006a ,**  Guidance on Sy stematic Planning Using the Data Quality Objectives Process , EPA QA/G
- 4 , U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2006b ,**  Data Quality Assessment: Statistical Methods for Practitioners , EPA QA/G
- 9 S , U.S. Environm ental Protection Agency
. http://www.epa.gov/quality **EPA 2006c ,**  Peer Review Handbook , U.S. Environmental Protection Agency
. http://www.epa.gov/peerreview/pdfs/prhandbk.pdf **EPA 2006d ,**  Pre - Dissemination Review Guideline s, U.S. Environmental Protection Agency
. http://intranet.epa.gov/quality/informationguidelines/pdf/pdr
- guidelines.pdf **EPA 2008a ,**  Quality

Policy , CI O Policy

2106 .0, Oct. 20, 2008 (“Quality Policy”) , U.S. Environmental Protection Agency
. http://www.epa.gov/irmpoli8/policies/21060.pdf **EPA 2008b ,**  Procedure for Quality Policy , CI O 2106 - P - 01 .0, Oct. 20, 2008 (“Quality Procedure”) , U.S. Environmental Protection Agency
. http://www.epa.gov/irmpoli8/policies/2106p01.pdf **EPA 2008 c**  **,** Guidance on Environmental Data Verification and Data Validation , EPA QA/G
- 8 , U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2009a,** EPA Records Management Policy , U.S. Environmental Protection Agency
. http://epa.gov/records **EPA 2009 b**  **,** Guidance on the Development, Evaluation, and Application of Environmental Models, Co uncil for Regulatory Environmental Modeling ( EPA/100/K
- 09 ) , U.S. Environmental Protection Agency
. http://www.epa.gov/crem/library/cred_guidance_0309.pdf
. **EPA 2009c**  **,** Guidance for Labeling Externally Validated Laboratory

Anal ytical Data for Superfund Use, OSWER No. 9200.1 - 85 ( EPA 540 - R - 08 - 005 ) , U.S. Environmental Protection Agency . www. epa .gov/superfund/policy/pdfs/EPA - 540 - R - 08 - 005 .pdfQAPP Handbook 100 December 2012 Agency Review Draft **EPA 2010 ,**  Technical Basis for the Lowest Concentration Minimum Reporting Level (L CMRL) Calculator, Office of Water (EPA 815
- R - 11 - 001) , U.S. Environmental Protection Agency
. http://water.epa.gov/drink **EPA 2012,** EPA Regional Guidance for the Preparation of Quality Assurance Program Plans (R9/3.1), U.S. Environmental Protection Agency
. http://www.epa.gov/region09/qa/pdfs/mngmt
-plan_guidance_2012.pdf **EPA 201 3**  **a ,**  Quality

Standard for Environmental Data Collection, Production, and Use by EPA Organizations, CI O 21 06 - S - 01.0, In Preparation (“Internal Standard”) , U.S. Environmental Protection Agency , http://www.epa.gov/quality **EPA 201 3**  **b ,**  Quality

Standard for Environmental Data Collection, Production, and Use by Non EPA (E xternal) Organizations , CI O 2106 - S - 02.0, In Preparation (“External Standard”) , U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EPA 2013 c**  **,** Guidance on Quality Management Plans, CI O Guidance QMP Handbook , In Preparation (“QMP Guidance”) , U.S. Environmental Protection Agency
. http://www.epa.gov/quality **EU 2007 ,**  Uncertainty Analysis Guidance , European Commission 2007 http://www.harmoni
- ca.info/toolbox/Model_Uncertainty/index.php **Hubaux, A. and G. Vos** , 1970 , Decision and Detection Limits for Linear Calibration Curves , Analytical Chemistry
. **Vol. 42** , No. 8, pp. 849 - 855. **IQG 2001**  . Information Quality Act Section 515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106
- 554; H.R. 5658)
. www.nrc.gov/

public
- involve/info
- quality/pl 106 - 554 .pdf **NRC 2007,** Models in Environmental Decision Making, Committee on Models in the Regulatory Deci si on Process, Board on Environmental Studies and Toxicology , Division on Earth and Life Studies. National Academic Press , Washington D.C. **Manno, J., R.Smardon, J.V. DePinto, E.T. Cloyd, and S. Del Grando** 2008. The Use of Models in Great Lakes Decision Making: An Interdisciplinary Synthesis. Randolph G. Pack Environmental Institute, College of Environmental Science and Forestry . Occasi onal Paper 16. **Skoog, D. A.** 1985 , Principles of Instrumental Analysis, third edition . Saunders College Publishing. Philadelphia, Pennsylvania. **Taylor, J. K.** 1987 , Quality Assurance of Chemical Measurements. Lewis Publishers. Chelsea, Michigan. **Ziliak, S .**  **T .**  2011 , Matrixx v. Siracusano and Student v. Fisher , Significance, **vol 8,** p 131 - 134.QAPP Handbook A - 1 Agency Review Draft **APPENDIX A**  **CHECKLIST OF QAPP ELEMENTS**  QAPPs should , at a minimum, address all elements detailed in this Handbook as it supports the Internal and External Standards
. In some cases, certain elements will not be appropriate for a particular project
. Elements that do not apply can be addressed with a simple s tatement of why the information is not relevant or with a cross reference to another approved document in which the information appears. This checklist shows the section number and title for each element with columns for :
 describing wh y the element is impo rtant;  indicating what might suffice for the element in a simple project;  listing what additional information might be necessary for the element in a more complex project;  noting whether the element was discussed in the QAPP;  noting where in the QAPP it was discussed; and  explaining why

an element was n o t included. This checklist is meant to be useful either as a tool for the developer of the QAPP to ensure all elements are included , or as a tool for reviewers who are checking to be sure that all elements are appropriatel y included. This checklist is organized by the four primary

steps in the project life cycle: Plan
- Do - Chec k
- Act and presented with steps 1 and 3 shaded, steps 2 and 4 un
- shaded for clarity
. The elements for Chapter 2 are presented first, then Chapter 3, and finally Chapter 4. T o facilitate use, the generation of new data (Chapter 2) is shown in black, the use of existing data (Chapter 3) in green, and the application to models (Chapter 4) in blue. In order to help di scuss the degree to which the QAAP elements need to be addressed, a simple case is contrasted to a more complex case; an example of the graded approach being used. These two cases are, b y nature of being examples, fairl y simplistic as the detail of documentation depends on the importance of the project, an exploratory study needing significantly

less documentation than one involving a proposed regulation.QAPP Handbook A - 2 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **2.2.1** Title, Version,

and Approval/ Sign
- off To meet the requirements of

C lause 7. 5.1 of the Standards for an approved QAPP prior to initiating work Single page with title and required signatures Revisions history tracking all changes made to the QAPP over time **2.2.2** Document

Format and Table of Contents To make it easy for the

project team and reviewers to find the information they need Table of contents and page numbers Document control format

**2.2.3** Distribution

List To be sure that everyone

who needs it has access to, and awareness of, the latest version of the QAPP Included on the title page or in the project organization section A separate section with contact information and indication of under what conditions each individual needs to be made aware of revisions to the QAPP **2.2.4** Project

Organization and Schedule To ensure all key project

personnel are aware of their responsibilities and the timeframe for completion Names of key project personnel and period of performance for project Organizational chart of all key project roles wi th names and Gantt or similar tracking chart of schedule **2.2.5** Project

Background, Overvie w , and Intended Use of Data To provide sufficient

information as a foundation for the project goals and to clarify the expected uses of the data General overvie w an d statement of how data and mo dels will be used in this project Detailed project history, references to other background documentation, regulatory basis for data use **2.2.6** Data/Project

Quality Objectives and Measurement Performance Criteria To ensure that the data or

mo dels collected or used in the project are of sufficient quality to support project decisions Project goals, data required to meet those goals, statement of what criteria the data must meet to be acceptable for project use 7 - step DQOs, MQOs for all DQIs, statement of tolerable error ranges for project decisions **2.2.7** Special

Training Requirements and Certification To be sure that the project

has qualified personnel to perform all necessary functions List of specialized training required in the project Matrix of special training needs, personnel who are qualified, and timeframe of qualifications **2.2.8** Documentation

and Records Requirements To document project

process, outcomes, and supporting quality information, to meet clause 7. 10 o f the Standards Statement declaring where project documentation will be stored, including, but not limited to: QAPP, data packages, assessment records, interim and final reports Detailed documentation and storage requirements per regulation or contractual oblig ation. These may be indicated in the project schedule as well.QAPP Handbook A - 3 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **2.3.1** Sam

ple Collection Procedure, Experimental Design, and Sam pling Tasks To define the way the data

will be collected and state whether it is a probability
based or judgmental design Define population of interest, sample design and rationale for that design, number/time/location and ty pes of samples to be collected As appropriate, discuss multistage sampling, randomization procedures, stratification rationale, or any details that supp ort the selection of samples **2.3.2** Sam

pling Procedures and Requirements To ensure that appropriate

sampling methods are selected to meet project needs and produce data of sufficient quality for project decision making Include or reference SOPs for sample collection, on
site preservation, and cleaning and decontamination of sampling equipment Include media
- specific sample collection procedures, safety issues involved in sample collection, sample container descriptions, etc. **2.3.3** Sam

ple Custody Procedures , and Documentation To ensure sample

authenticity and to avoid sample loss Describe sample documentation and handling procedures and include a sample chain
- of

- custody form List US D OT regulations and how they will be met, explain sample numbering schema, de fine procedures for introducing PE samples **2.3.4** Analytical

Methods Requirements and task Descriptions To be sure appropriate

measurement methods exist to achieve project quality objectives Describe measurement techniques, such as counting, visual discrimination, or analytical methods Details of analytes to be measured, analytical method, SOPs, data
-package requirements, analytical laboratory contact information, and MQOs **2.3.5** Quality Control

Requirements To provide confidence that

the project dat a are of suitable quality to be used for their intended purpose Number and types of QC samples to address sources of measurement error Table of sources of potential measurement error and the corresponding QC samples that will be used to address those uncertainties **2.3.6** Instrument/

Equipment Testing Calibration and Maintenance Requirements , Supplies and Consumables To avoid poor instrument

performance that could im pact project data quality Document roles and responsibilities and procedures for sample collection and measurement instrument inspection, calibration, and maintenance Include instrument measurement uncertainty, traceabil ity of calibration equipment, and project
specific schedule for inspection, calibration, and maintenance **2.3.7** Data

Management Requirements To ensure data integrity Describe data handling from generation, to use, to final storage; include copies of data entry forms, reports and description of databases; specify any special requirements for data such as CBI or hardware/software Include SOPs for data management, specify project personnel’s roles and responsibilities regarding data management, describe how metadata will be gathered and stored with project dataQAPP Handbook A - 4 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **2.4.1** Technical

Sy stems Assessments To determine if the field

sampling team, laboratory, or analysts have sufficient technical capabilities to generate data of appropriate quality for project use If project
- specific needs are minimal, noting accreditation to an appropriate technical system standard may be sufficient ; note that if any project
specific needs are more stringent than the standards, then project
- specific assessments should be conducted De scribe plans for assessments of field sampling, on
- site and off
site laboratory analyses

**2.4.2** Performance

Audits of Measurement and Analytical Sy stems To directly test that the

measurement performance is adequate for project purposes Document plans and acceptance criteria for split samples and PE samples, if appropriate Specify schedule, analyte selection, traceability, spiking levels, and sample ty pes (matrix, direct, single
blind, or double
- blind), and include the sources and estimated costs for these samples **2.4. 3**  Surveillance of

Operations To verify that project

activities are conducted as planned State when surveillance will occur (under what conditions or by set timeframe), how

it will be conducted, how feedback will be provided and incorporated, and if surveillance leads to a tem porary or permanent work stoppage, address how that will be handled Provide details of the triggering events for surveillance assessments, an SOP for their conduct, a list of who will be notified of any non - conformances ob served during the assessments, and how the surveillance assessments will be documented **2.4. 4**  Audits of Data

Quality To be sure that QC data are

used to support data quality, and to determine if the data are replicable Define the schedule (based on timeframe or triggering events) and scope for audits of data quality Provide an SOP for conducting audits of data quality **2.4. 5**  Interim

Assessments of Data Quality To be sure data collection is

proceeding according to plan Include statement encouraging project team members to alert management if they sense anything isn’t going quite right Indicate points during the project timeline at which interim assessments of data quality will be conducted; state who will be responsible for their conduct and how they will be documentedQAPP Handbook A - 5 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **2.4. 6**  Qualitative and

Quantitative Com parisons to Acceptance Criteria To determine if the project

quality objectives are met State how comparisons to qualitative and quantitative MQOs will be evaluated; describe other criteria (e.g., publication in a peer
reviewed journal) that might be important for the project Set specific comparison methodology for each DQI, such that there will be a definitive “pass/fail” for each; describe contingencies in case of “f

ail” **2.4. 7**  Evaluation of

Unconventional Measurements To be sure that any unusual

laboratory methods are fully documented and understood If no unconventional measurement methods will be/were used, just state that here Describe the reason for using unconve ntional measurement methods, provide SOPs for the ir im plementation and how efficacy will be assessed **2.4. 8**  Evaluation of

Unconventional Monitoring Projects To be sure that any unusual

sample collection methods are fully documented and understood If no unconventional sample collection methods will be/were used, just state that here Describe the reason for using unconventional sample collection methods, provide SOPs fo r their im plementation and how efficacy will be assessed **2.5.1** Data

Verification and Validation Targets and Methods To determine if data have

met the project quality objectives Provide a standard data verification and validation method or procedure that has been reviewed to ensure it meets project needs Develop project
- specific verificatio n and validation schema that incorporate QC data and metadata **2.5.2** Quantitative

and Qualitative Evaluations of Usability To ensure that project

decisions are supported by data of sufficient quality for project needs State who will be part of the evaluation of data usability, how it will be conducted and documented Define contingencies for any issues that may be identified during the evaluation of data usability **2.5.3** Potential

Limitations on Data Interpretation To ensure that the data are

not stretched beyond their appropriate use Describe what actions will be taken if project data are deemed unusable for their intended project purpose Describe how any limitations will be documented and stored in the project metadata **2.5.4** Reconciliation

with Project Requirements To determine if project

requirements have been met Clearly state how the data verification, validation, and usability results will be used to determine if project requirements have been met; describe how

the five steps of the DQA proc ess will be conducted State specific exceptions to statistical signficance that will be overturned in favor of “practical signficance” should they occur; define the steps to be taken for contingencies if data do not support requirements **2.5.5** Reports to

Management To document project

outcomes Define schedule an d content for reports to management Provide report templatesQAPP Handbook A - 6 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **3.2.1** Title, Version,

and Approval/ Sign
- off To meet the requirements of

C lause 7. 5.1 of the Standards for an approved QAPP prior to initiating work Single page with title and required signatures Revisions history tracking all changes made to the QAPP over time **3.2.2** Document

Format and Table of Contents To make it easy for the

project team and reviewers to find the information they need Table of contents and page numbers Document control format

**3.2.3** Distribution

List To be sure that everyone

who needs it has access to, and awareness of, the latest version of the QAPP Included on the title page or in the project organization section A separate section with contact information and indication of under what conditions each individual needs to be made aware of revisions to the QAPP **3.2.4** Project

Organization and Schedule To ensure all key project

personnel are aware of their responsibilities and the timeframe for completion Names of key project personnel and period of performance for project Organizational chart of all key project roles with names and Gantt or similar tracking chart of schedule **3.2.5** Project

Background, Overvie w, and Intended Use of Data To provide sufficient

information as a foundation for project goals and to clarify expected uses of the data General overvie w and statement of how data and mo dels will be used in this project Detailed project history, ref erences to other background documentation, regulatory basis for data use **3.2.6** Data/Project

Quality Objectives and Measurement Performance Criteria To ensure that the data or

mo dels collected or used in the project are of sufficient quality to support pr oject decisions Project goals, data required to meet those goals, statement of what criteria the data must meet to be acceptable for project use 7 - step DQOs, MQOs for all DQIs, statement of tolerable error ranges for project decisions **3.2.7** Special

Training Requirements and Certification To be sure that the project

has qualified personnel to perform all necessary functions List of specialized training required in the project Matrix of special training needs, personnel who are qualified, and timeframe of qualifications **3.2.8** Documentation

and Records Requirements To document project

process, outcomes, and supporting quality information, to meet clause 7. 10 of the Standards Statement declaring where project documentation will be stored, including, but not limited to: QAPP, data assessment records, interim and final reports Detailed documentation and storage requirements per regulation or contractual obligation. These may be indicated in the project schedule as well.QAPP Handbook A - 7 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **3.3.1** Proposed Data

Source Originator and Publication Information To provide documentation

and defensibility of data considered for inclusion in the project Include name and date of data originator/origination Publication source and peer review history

**3.3.2** Data Format

and Accessibility To determine if the data are

accessible for project use Describe data format, data fields, and accessibility issues, if any Provide SOPs for accessing data **3.3.3** Establishment

of Acceptance Criteria To set specific quality

requirements for d ata brought in for project use Establish the MQOs (qualitative or quantitative, when possible) required to meet project objectives Discuss contingencies and possible trade
- offs if MQOs are not directly achievable with existing data **3.3.4** Sam

ple Data Coll ection Methodology To be sure data are

appropriate for project purposes State whether data were collected using a probability
- based or judgmental sampling design Explain the sample design and collection procedures and their implications for project use, such as whether it is reasonable to infer wider meaning beyond the original intent of the data collection **3.3.5** Quality

Program and Quality Assurance Procedures Used by Data Originator To document the quality of

the existing data and to determine if the quality program is sufficient to support the use of th ese data for project purposes Reference the quality program under which data were collected; if unknown, state the limitations on using data of unknown quality Describe results of QC samples included wi th existing data, data originator notes or reports on data collection, analyses, and limitations on interpretation **3.3.6** Documentation

of Sample Quality Assurance Procedures To facilitate the conversion

of existing data to a usable format without error. A simple statement or description of how this was achieved Documentation of the methods used and procedures taken to ensure accuracy in conversion to suitable format **3.3.7** Data

Management Requirements To ensure data integrity Describe data handling from generation, to use, to final storage; include copies of data entry forms, reports and description of databases; specify any special requirements for data such as CBI or hardware/software Include SOPs for data management, specify project personnel’s roles and responsibilities regarding data management, describe how metadata will be gathered and stored with project dataQAPP Handbook A - 8 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **3.4.1** Qualitative

Com parisons to Acceptance Criteria To be sure existing data

meet the qualitative accept ance criteria for project use Describe how the project will assess how

w ell data represent the underlying population of interest, and how comparability among data used in the project will be assessed Include a table with a discussion of all qualitative asp ects of data quality with MQOs for representativeness, completeness, and comparability **3.4.2** Quantitative

Com parisons to Acceptance Criteria To be sure the existing data

meet the quantitative acceptance criteria for project use Document the precision, bias, and sensitivity required to meet project objectives Include a table with specific MQOs for the quantitative DQIs (precision, bias, sensitivity, and perhaps completeness) with discussion of contingencies and possible trade
- offs between MQOs if the dat a do not fully meet the requirements **3.4.3** Assessments to

Other Criteria To be sure there are n o t

other issues that will inhibit the likelihood of a successful project State any constraints under which the project must function to be successful (e.g., timeframe, cost, attainment of specific goals) Describe the mechanisms in place to ensure that these constraints are met **3 .4.**  **4** Interim

Assessments of Data Quality To be sure data collection is

proceeding according to plan Include statement encouraging pr oject team members to alert management if they sense anything isn’t going quite right Indicate points during the project timeline at which interim assessments of data quality will be conducted; state who will be responsible for their conduct and how they will be documented **3 .4.**  **5** Evaluation of

Unconventional Measurements To be sure that any unusual

laboratory methods are fully documented and understood If no unconventional measurement methods will be/were used, just state that here Describe the reason for using unconventional measurement methods, provide SOPs for their im plementation, and describe how

their efficacy will be/was assessed **3.4.6** Evaluation of

Unconventional Monitoring Projects To be sure that any unusual

sample collection methods are fully d ocumented and understood If no unconventional sample collection methods will be/were used, just state that here Describe the reason for using unconventional sample collection methods, provide SOPs for their im plementation, and describe how

their efficacy will be/was assessedQAPP Handbook A - 9 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **3 .5.1**  Data

Verification and Validation Targets and Methods To determine if data have

met project quality objectives Provide a standard data verification and validation method or procedure that has been reviewed to ensure it meets project needs Develop project
- specific verification and validation schema that incorporate QC data and metadata **3 .5.2**  Quantitative

and Qualitative Evaluations of Usability To ensure that project

decisions are supported by data of sufficient quality for project needs State which members of the project team will be part of the evaluation of data usability, how it will be conducted and documented Define contingencies for any issues that may be identified during the evaluation of data usability **3 .5.3**  Potential

Limitations on Data Interpretation To ensure that the data are

not stretched beyond their appropriate use Describe what actions will be taken if project data are deemed unusable for their intended project purpose Describe how any limitations will be documented and stored in the metadata surrounding the project data **3 .5.4**  Reconciliation

with Project Requirements To determine if project

requirements have been met Clearly state how data verification, validation, and usability results will be used to determine if the project requirements have been met; describe how

the five steps of the DQA process will be conducted State specific exceptions to statistical signficance that will be overturned in favor of “practical signficance” should they occur; def

ine the steps to be taken for contingencies if the data do not fully support project requirements **3 .5.5**  Reports to

Management To document project

outcomes Define schedule and content for interim and final reports to management Provide report templates
**4.2.1** Title, Version,

and Approval/ Sign
- off To meet the r equirements of

C lause 7. 5.1 of the Standards for an approved QAPP prior to initiating work Single page with title and required signatures Revisions history tracking all changes made to the QAPP ove r time **4.2.2** Document

Format and Table of Contents To make it easy for the

project team and reviewers to find the information they need Table of contents and page numbers Document control format

**4.2.3** Distribution

List To be sure that everyone

who needs it has access to, and awareness of, the latest version of the QAPP Included on the title page or in the project organization section A separate section with contact information and indication of under what conditions each individual needs to be made aware of revisions to the QAPPQAPP Handbook A - 10 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **4.2.4** Project

Organization and Schedule To ensure all key project

personnel are aware of their responsibilities and the timeframe for completion Names of key project personnel and period of performance for project Organizational chart of all key project roles with names and Gantt or similar tracking chart of schedule **4.2.5** Project

Background, Overvie w, and Intended Use of the Model To provide sufficient

information as a foundation for project goals and to clarify the expected uses of the data General overvie w and statement of how data and mo dels will be used in this project Detailed project history, references to other background documentation, regulatory basis for data use **4.2.6** Data/Project

Quality Objectives and A cceptance Criteria To ensure that data f or mo dels collected , or used in the project , are of sufficient quality to support project decisions Project goals, data required to meet those goals, statement of what criteria the data must meet to be acceptable fo r project use 7 - step DQOs, MQOs for all DQIs, statement of tolerable error ranges for project decisions **4.2.7** Special

Training Requirements and Certification To be sure that the project

has qualified personnel to perform all necessary functions List of specialized training required in the project Matrix of special training needs, personnel who are qualified, and timeframe of qualifications **4.2.8** Documentation

and Records Requirements To document project

process, outcomes, and supporting quality information, to meet clause 7. 10 of the Standards Statement declaring where project documentation will be stored, including, but not limited to: QAPP, data packages, assessment records, interim and final reports Detailed documentation and storage requirements pe r regulation or contractual obligation. These may be indicated in the project schedule as well.

**4.3.1** Data

Requirements for Model Input To ensure that appropriate

data are available for use in the model Describe the type, quantity, and quality of data required for use in the model, and how it will be identified or collected Specify data collection or production and analysis requirements for the model input data (per C hapters 2 or 3 of this QAPP Guidance) **4.3.2** Data

Management Tasks To ensure the integrity of

data used in the project Discuss how data will be collected, produced, managed, and stored Include SOPs for data management **4.4.1** Model

Evaluation Methods and Activities To know whether or not the mo del and its results are of sufficient quality to meet project objectives Describe how the project will evaluate that the model is based on documented science, appropriate data, performs adequately, and meets project objectives Include specific model evaluation plans throughout the project and mo del’s lifecycle, including factors that would trigger model mo dificationQAPP Handbook A - 11 Agency Review Draft **Section Number**  **Title Why Important**  **Simple Case Complex Case**  **4.4.2** Evaluation of

the Model To be sure the model

functions as needed for project purposes Documentation of how the mo del has been and will continue to be assessed for its ability to provide output that meets project objectives Calibration and internal quality check logs, model change/development logs, evaluation criteria and identification of responsible staff **4.4.3** Model Output

Sensitivity and Uncertainty Analysis To determine if the model is

providing sufficient certainty to reach a decision with the desired level of quality List anticipated outputs of sensitivity and uncertainty analyses, how

they will be documented, and any criteria for their acceptability Com puter code for the uncertainty and sensitivity analyses, documentation of the criteria that would lead to refinement of the model via the path identified in the sensitivity analysis **4.5.1** Peer Review To ensure the validity of the science behind the development of the model Peer Review by colleagues familiar with the problem Formal Agency Peer Revie w with documented charge and response **4.5.2** Description of

Model Documentation To provide defensibility and

reproducibility Description of how model selection or development, mo difications, input data, results, sensitivity and uncertainty analyses, and evaluation will be documented SOPs for document control and archiving of programming code, software, and related manuals **4.5.3** Specifications

for Model Maintenance and User Support To ensure the model doesn’t

cease to function properly during the project lifecycle Simple statement of the intent to maintain the model throughout the project lifecycle Specific schedule for model mai ntenance and SOP or description of user support to be provided throughout the project lifecycle **4 .5.**  **4** Reports to

Management To document project

outcomes Define schedule and content for interim and final reports to management Provide report templatesQAPP Handbook B - 1 December 2012 Agency Review Draft **APPENDIX B DATA QUALITY INDICATORS**  The choice of MQOs for selection of Data Quality Indicators ( DQI s ) is critical to evaluating and, in some cases, controlling environmental data quality
. The criteria that will be used to gauge measurement performance should be determined and documented for each matrix, analy tical group, concentration level, and anal yt e, if applicable. DQIs relate to the parameters of precision, accuracy /bias, representativeness, comparability, sensitivity, and completeness. Itis extremel y important to recognize that data quality is a function of both the sampling and measurement processes; it is not so lely equated to the quality of individual analytical results
. To simplify the way data quality

is examined, and to facilitate communication about data
- quality attributes, the following DQIs important to environmental studies will be defined: precision, bias, representativeness, comparability , completeness, and sensitivity (PBRCCS)
. These six DQI s used to be referred to b y the acronymPARCCS, with the "A" in PARCCS referring to accuracy instead of bias
. Accuracy is actually comprised of random error (prec ision) , and sy stematic error (bias), and these indicators are discussed separately and hence the “A” has been replaced b y “B” . The goal of this appendix is to familiarize the reader with the ty pes of DQIs, how each can be used to assess data quality and us ability , and how they are measured
. DQIs for precision, bias, and sensitivity can be defined and MQOs measured in quantitative terms; MQOs for representativeness, comparability , and completeness can be defined and measured in both qualitative and quantit ative terms
. Establishing performance criteria for representativeness is a principal element in setting goals for the sampling design
. Establishing performance criteria for sensitivity sets quantitative goals for the quality of data generated in the anal ytical measurement process
. Establishing performance criteria for precision and bias potentially

sets quantitative goals for the quality of data generated in both the sampling and anal ytical measurement process
. **Table B**  **- 1**  **Relationship Between**  **Quality Terms** DQOs or PQOs Qualitative and quantitative quality objectives for project conclusions or decisions . The

decision to call these Data Quality Objectives ( DQOs ) or Project Quality Objectives ( PQOs ) depends on the organization’s preferences. DQIs These are the indicators of data quality attributes
. MQOs or MPCs Acceptance thresholds or goals for the data, usually based on individual DQIs . The decision

to call these Measurement Quality Objectives ( M QOs ) or Measurement Project Criteria ( MPCs ) depends on the organization’s preferences.
M easurement Quality

Objectives (M QOs ) should be developed as an integral part of the sampling and anal ys is design generated during s ys tematic planning (e.g., in Step 7 of the DQO process) and see also Annex C of the two Standards
. MQOs are not s ynonymous with project DQOs. DQOs establish the full set of specifications for the design of the data collection effort
.QAPP Handbook B - 2 December 2012 Agency Review Draft The design ty

picall y incorporates and specifies requirements for total variability
. These requirements are used to e stablish performance criteria, stated as MQOs ( MPCs ) , for significant components of the total variability
. It is emphasized that sampling and analy tical procedures should be selected after the performance criteria have been established
. The relationship am ong these quality

terms (DQOs/PQOs, DQIs, and MQOs/MPCs) is presented in Table B - 1 . The historical focus of DQIs has been on the laboratory measurement processes, but, as discussed later, DQIs can be chosen to capture the effects of the full measurement s y stem, sampling design, and sample collection processes, all of which affect the overall quality of study data
. Sections B.3 through B.9 detail the DQIs and specific issues relating to each
. Table B - 2 offers a summary of the definitions for data quality attributes commonly monitored with DQIs along with brief descriptions of methods for determination of those attributes
. **Table B**  **- 2**  **D ata**  **Q uality**  **I ndicator**  **s DQI**  **Definition Examples of**  **Determination** Precision An evaluation of agreement among

replicate measurements of the same property under similar conditions; also referred to as random error or measurement variability and usually expressed as standard deviation, variance, percent difference, or range, in either absolute or relative term s Overall project precision is measured by collecting data from co l located field duplicate (or replicate) samples. Precision specific to the laboratory is measured by analyzing laboratory duplicate (or replicate) samples
Bias T he systematic or persistent distortion of

a measurement process resulting in error in one direction Measurement of materials with a known concentration (e.g. performance evaluation or reference materials), analy sis of matrix spikes , or the use of laboratory control samples Accuracy A measure of the closeness of an

individual measurement to a known or reference value; includes a combination of random error (precision) and systematic error (bias) components of both sampling and analytical operations Replicate analysis of a reference material or re
analyze a sample to which a material of known concentration or amount of pollutant has been added; usually expressed either as percent recovery or as a percent bias Representative ness A qualitative measure of the degree to

which data accurately and precisely represent a characteristic of a population parameter Evaluation of whether a sample that is collected and then processed and sub
- sampled by the laboratory is proportionately represe ntative of some predefined population characteristic or property. As such, representativeness is an “objective
- defined” parameter (e.g. total concentration versus dissolved concentration versus bioavailable concentration) Com parability A qualitative term describing the degree

to which different processes, methods , or data agree or can be represented as similar. It describes the confidence that two datasets can contribute to a common analysis and interpolation
. Comparability criteria must be determined for each matrix, analytical group , concentration level, and analyte (if possible). A comparison of the output of two sediment transport models via sensitivity analysis
. Or comparison of the sample collection methods, analytical procedures, holding times, stability issues and QA protocols . One study with results in ug/L is not necessarily comparable to another with results in ppb. A similar argument exists between wet and dry weight comparisionsQAPP Handbook B - 3 December 2012 Agency Review Draft **DQI Definition**  **Examples of Determination**  Com pleteness An evaluation of the amount of data

needed to be obtained from a measurement system; expressed as a percentage of the number of measurements that should have been collected or were planned to be collected E valuation of the number of measurements needed to make a determination of the project results and comparison of this to the number of samples planned to be collected Sensitivity T he capability of a method or instrument

to discriminate the parameter of interest at the level of interest. Terms sometimes used to describe sensitivity include M ethod D etection limit (MDL), L imit of D etection (LOD), and L imit of Q uantitation (LOQ) The measurement responses representing different levels or amounts of the variable of interest , MDL study , and verification of LOD

**B.1 SOURCES OF MEASUREME**  **NT UNCERTAINTY** Before discussing details of DQI calculations and uses, a presentation of the framework used in this document for examining error sources is necessary
. Consideration of all the sources of error is critical as specific DQIs can be established to address specific sources of error
. Total study error is a measure of the uncertainty in a metric such as a site mean concentration caused b y the combination of all error sources in the study design
. The term “total” is relative to the overall boundary conditions (spatial or temporal boundaries) for which a decision will be made (sometimes referred to as the decision unit) or study conclusions drawn, based on the relevant metric
. The se boundaries are defined in the DQO process and may include the overall population of interest, as well as specific subpopulations for which a decision will be made
. A sampling unit is the portion of the phy sical environment from which one or more samples may be taken, resulting in measurements appropriate for an intended use
. The term "error sources" refers to any factors that increase uncertainty in a measured value
. Generall y speaking, these error sources are the result of natural variability in the sampled media and uncertainty associated with the sampling and measurement process . The framework for examining study errors will also serve as a context for clarifying the relationship among the total study error, DQOs, and MQOs. The contribution to total error from within
- unit sources is largely a function of the boundary conditions for the study

at hand, the inherent variability of the characteristic of interest within the unit , the difficulties associated with obtaining a specim en from the unit, and the error associated with the measurement process itself
. **B.2 ESTABLISHING MQO**  **s IN THE CONTEXT OF DQ**  **O s**  One way to employ

DQIs is as a means of specifying MQOs which, if achieved, will provide an indication that the resulting data ar e expected to meet the DQOs
. This is not to say that simply achieving MQOs for the measurements ensures that the DQOs are met
. Ty pically it is more important to ensure that an adequate number of samples, with appropriate sample support, are collected to represent the population of interest
. If this is the case, then ensuring that MQOs were achieved provides an indication that the measurement quality will be adequate for the intended use . Used in this way , DQI s provide a metric against which the performance of a program can be measured during the implementation and/or assessment time frames
.QAPP Handbook B - 4 December 2012 Agency Review Draft During the design phase, the ty pe and number of samples required to achieve DQOs and the way in which these samples should be optimally allocated across space and time are developed
. Guidance on Choosing a Sampling Design for Environmental Data Collection , EPA QA/G
- 5S, ( EPA 200 2 a) provides guidance on alternative design approaches
. For statistically
- based sampling designs, an estimate of total study error is required to d etermine the appropriate number of samples required to achieve a certain performance goal, In some cases, historical data and information are available, but were collected with older, or different, sampling or anal ytical methods than are currently considered (comparability)
. In other cases it may be advisable to derive an estimate of variability using other information on the background of the measured variable. Usually

this involves the use of statistical theory combined with practical knowledge. To determine the potential impact of using newer methodologies on total study accuracy (precision and bias) , it is useful to see how much the error sources of variability contribute to the total study

variability
. If the newer methods are more accurate , specific MQOs can be established based on the expected improvements in laboratory performance, and these improvements can be factored into the design equation
. Itis emphasized that the performance goals of precision (and the rest of the MQOs) be considered relative to action levels for each analy te . For example, i f the newer methods are less accurate, but the concentrations are well above (or below) their action levels such that a decrease in accuracy or loss of precision will not affect the ability to m ake an appropriate decision, then it may still be optimal to use them
. If heterogeneity is problematic , then obtaining a larger number of samples would increase coverage (representativeness) to better represent the population of interest and account for be tween - unit variability. A practical approach to accomplish this is to first determine the performance required for the DQI, based on an analy sis of the relative impact of a specific error source or quality attribute on total study

design
. This will be bas ed on an understanding of what method or procedure precision levels are tolerable, given a particular design (typ e, number, and allocation of samples) under consideration. The design process involves evaluating tradeoffs between the sampling aspects and me asurement aspects and should be aimed at achieving an optimal combination of the two relative to the DQOs or project objectives
. If laboratory

or method defaults are not adequate or if alternatives are much less expensive, then MQOs should be specified an d a new agreement reached with the laboratory performing the anal ys es (or a new instrument or method selected) to meet or exceed these more stringent requirements
. If aimed at the anal ytical laboratory , this may take the form of a special anal ytical reques t or a performance
- based method contract
. If aimed at the use of an instrument or method, it may take the form of a new standard operating procedure designed to ensure that the MQO is achieved such as SOPs for replicating X - ray fluorescence (XRF ) measurements to achieve a precision requirement
. MQOs should be driven by the DQO process and should alway s be defined before listing laboratory or method "default" limits in a QAPP
. This is especially important when the project does not require levels as stringent as the laboratory ’s default limits to be adequate for the intended use
. For example, if default requirements for sensitivity are far better than required to support a particular decision, project
- specific, less restrictive requirements can be stated in th e QAPP .QAPP Handbook B - 5 December 2012 Agency Review Draft **B .3**  **PRECISION** Precision is a DQI upon which statistically designed sampling is based
. For this reason, precision is among the most important DQIs in environmental projects
. Precision is the measure of agreement among repeated measurements of the s ame property under identical or substantially similar conditions
. High precision refers to close agreement among repeated measures, whereas low precision refers to poor agreement among repeated measures
. For environmental studies, the property measured is ty

pically a concentration of a contaminant, but any phy sical measurement has error; therefore, the precision of that measurement can be examined
. Random errors or fluctuations in the measurement process alway s result in some range of values of these repeat ed measurements . P recision is a quantitative indicator of the dispersion generated from these random errors. Precision is estimated by

using some form of replication (e.g., collocated field duplicates, field splits, laboratory

splits, matrix spike duplicat es, instrument replicates) followed b y calculation of a DQI based on the replicate measurements
. Estimates of precision provide a measure of agreement among replicate anal ys es; MQOs for precision set tolerable limits of imprecision for a study
. One of the most common precision DQIs are the summary statistics range for duplicate measurements and standard deviation for multiple measurements
. One nice feature of these statistics is that their units are the same as the individual measurements, and they are rel ativel y eas y to calculate. Another widely used measure is relative percent difference, usually used when only two samples are taken. **B .4**  **BIAS** Bias is sy stematic or persistent distortion of a measurement process that causes errors in one direction
. Bias may originate from sources such as calibration errors, response factor shifts, unaccounted
- for interferences, or sample contamination
. The sample itself may generate real or apparent bias caused b y a matrix effect or variation in phy sical properties such as particle size
. DQIs for precision are quantitative indicator s , which reflect the magnitude of sy stematic error resulting from these effects
. Bias can be in the positive ( a false positive or elevated signal/concentration) or negative ( a false negative or su ppressed signal/concentration) direction from the true value. The difference between the measured and known or expected result is a DQI for bias
. For spikes and reference materials, bias is most conveniently expressed as a fractional or percent comparison of the measured result to the expected result
. Relative bias is also a common indicator, which indicates both the magnitude and direction (positive or negative) of the bias
. Expected results are based on the known properties of the QC sample
. The expecte d result is usually the true value derived from a standard or a theoretical value and documented in the project’s QAPP . measured result
- expected result relative bias = expected resultQAPP Handbook B - 6 December 2012 Agency Review Draft Relative bias is the ratio of the difference between the measured and expected results t o the expected result
. Relative bias is sometimes used to describe bias in terms analogous to relative precision
. For applications of bias with a known contaminant, a commonly used measure of bias is percent recovery , which is simply the percent of the expected result that is measured
. Percent recovery calculated from matrix spikes is the difference between the spiked and unspiked sample results divided by

the spiking amount
. A completely unbiased result thus has a recovery of 1 (or 100%)
. Minor short
- term problems that do not create a systematic bias are best reflected in precision DQIs (random errors)
. For example, if an anal ys t makes an error on a dilution as part of the sample preparation step, that mistake will bias the individ ual result . H owever, this sort of error is part of the overall precision in the measurement process
. Because the anal ysis of QC samples (see Appendix C) includes the entire measurement step, the precision of the measurement process is also imposed on resul ts intended for estimating bias. For this reason, estimates of bias primarily serve as a quality

control function during implementation of a project plan. During the planning phase, procedures put in place to minimize the potential for bias and the use of estimates of bias to verify the procedures should have the desired effect of keeping bias to a minimum and assessing the potential effects on project objectives
. Procedures for estimating bias are also valuable tools for ensuring comparability of data
. Co mpleteness of a data set also has the potential to impact bias
. If a data set is incomplete, any sy stematic trend to the missing data may cause bias in estimates of population parameters based on th is faulty data set
. **B .5**  **A CCURACY**  Accuracy is a measure of the overall agreement of a measurement to a known value
. In a limiting case where random errors are very tightl y controlled, bias dominates the overall accuracy
. In general, however, both precision and bias contribute to accuracy
. A measurement result with zero bias may

not be accurate if the measurement process is not precise
. Figure B - 1 demonstrates how different combinations of precision and bias can contribute to accuracy
. The true mean value, indicated b y the dashed line, is 10 .0 and the red line represents a hypothetical action level of 12.0 that might be used as a “bright line” or limit for an action
. Diamond sy mbols indicate the observed measurements
. Part A of Figure B - 1 shows the least desirable situation of positive bias (a large shift in the positive direction) and low precision (a large spread in the results)
. The accuracy and precision of these data are poor . Part B of Figure B - 1 shows the case of low bias (no apparent shift in a positive or negative direction) and low precision
. Any one of the measurements is not accurate, but an average of all the measurements would be accurate relative to the true mean
. Part C of Figure B - 1 shows data with positive bias and high precision
. The accuracy of these data is poor
. In this case, random errors are well controlled, but a s ys tematic error limits the accuracy

of each individual measurement
. Part D of Figure B - 1 shows the case of low bias and high precision. Each individual measurement is accurate. Notice that in C and D the DQI of precision is critical for making a correct decision relative to the action level
. Itis obvious from this visual example that both precision and bias be considered relative to the action level in order to assess whether the DQI has a significant impact on the ability to make a correct decision. Whenever an action level is “close” to the measured data from the site (relative to the dispersion of the data), the more critical the DQIs are
.QAPP Handbook B - 7 December 2012 Agency Review Draft **Figure B**  **- 1**  **Influence of Bias and Precision on Accuracy B.6**  **REPRESENTATIVENESS** Representativeness , as it was defined b y ANSI/ASQ E4
- 2004 (ANSI 2004) , is "The measure of the degree to which data accuratel y and precisel y represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an env ironmental condition." This definition of representativeness encompasses issues at both the micro
- and macro
- scale b y addressing both how well measurements taken within a sampling unit reflect that unit (parameter variations at a sampling point) and the de gree to which measurements from a set of sampling units represent (allow you to make inferences about) the population of interest (accuratel y and precisel y represent a characteristic of a population). Central to representativeness is assurance that both th e sampling and measurement processes are free from known biases
. Developing a clear understanding of the "population" that is the subject of an experiment or investigation is the key to assessing representativeness
. The characteristics of the population in clude the subject's identity or class (e.g., the particular property that is to be measured), the spatial distribution of the property, and in some cases, the temporal characteristics of the property . Step 4 of the DQO process ( EPA 2006a) explicitly focuse s on establishing a clear definition of the population of interest and any subpopulations that data should adequatel y represent to support decisions at the desired scale
. Representativeness is usually considered a qualitative term that does not lend itself to measurement by

an MQO .QAPP Handbook B - 8 December 2012 Agency Review Draft Instead, sampling precision and bias indicators related to the desire to represent environmental phenomenon at specified spatial and temporal scales are used to control the quantitative aspects of representativeness
. Representativeness was established as a DQI as a result of the recognition that characteristics of interest to environmental problems are heterogeneously distributed in space and time within the environment, and that careful attention should be paid during planning and implementation of a study to ensure that a set of samples adequatel y mirrors or reflects the characteristics of interest
. Project managers and decision makers want assurance that the results of a particular data collection effort are not bias ed in any known way by the sampling or anal ysis design
. Such a bias, combined with the inherent heterogeneity in the environment and uncertainty in the measurement process, could result in an incorrect conclusion
. Lack of representativeness can have a dire ct impact on the ability to make the correct decision when rel yi ng on data
. To ensure representativeness, careful attention during the entire life cycle of a project is required
. The sections on precision and bias covers in detail some of the more quantit ative elements of representativeness
. Central to representativeness is assurance that both the sampling and measurement processes are free from known biases
. Representativeness is widely used in a less precise manner in the environmental community
. For example, representativeness is a word commonly

used to mean:
 there is an absence of biasing forces;  it is a miniature or replica of the population;  it is a ty

pical or ideal case;  there is a wide coverage of a population;  it permits good estimation;  it is goo d enough for the purposes of the study; or  a statistically
- based sampling method was used
. While a number of these are indeed characteristics of a representative study, a more careful definition will better delineate suitable indicators of representativeness . The concept of representativeness is called out in a number of EPA regulatory programs . For example, the Resource Conservation and Recovery Act (RCRA) talks about “...expected to exhibit the average properties of the universe or whole” (RCRA CFR 260 .10)
. The Toxic Substances Control Act (TSCA) refers to “...at locations representative of the air entering the abatement site ” (TSCA 40 CFR 763). In the air programs we see a discussion of representativeness: “...should be selected on the basis of spatial and temporal representativeness” (40 CFR 51 Appendix W). The waste water program simply state s , “...samples should be representative of daily

operations ” (40 CFR 403.12(b))
. While these statements are lacking in a rigorous definition, they

provide some understanding of the importance of this indicator to EPA programs .QAPP Handbook B - 9 December 2012 Agency Review Draft The process involved in obtaining representative samples includes planning, implementation, and assessment . In addition to sample design, careful attention should be paid to the sample collection, measurement and anal ys is processes
. Representativeness also has relevance in the world of laboratory research studies and experimental design
. For a discussion of these issues, refer to texts on experimental design, including Box , Hunter, and Hunter (1978) and Cochran and Cox (1957). To understand the importance and scope of this DQI, imagine just these few examples where representativeness is not achieved and consider the potential consequences:
 surface water samples are taken from the stagnant water nearest the shore in a calm pool, and are then meant to represent the characteristics of the fast
- flowing river;  soil samples are collected using a technique that preferentiall y selects the smallest particles and omits the larger particles, thus under
- representing those with lower surface area proportionatel y, and therefore lower levels of externally adhering contaminants; or  a survey

of public perception to off
- shore oil drilling conducted entirely in Ohio, Kansas, and Nevada, and presented as national opinion. Representativeness, as a DQI, is most relevant when viewed in the context of the data’s intended use . Representativeness between sampling units and within sampling units should be considered, and qualitative MQOs for both should be established and documented in the QAPP. **B.7 COMPARABILITY**  Comparability

is the qualitative term that expresses the confidence that two datasets can contribute to c ommon interpretation and anal ys is of the parameter or matrix of interest
. Quantitative measures of comparability are also available involving statistical tests that measure the similarity

or difference between two or more datasets
. Comparability should be carefull y evaluated in order to establish whether two datasets can be considered equivalent in regard to the measurement of a specific variable or groups of variables (EPA 1997, EPA 2010a). Comparability

is a very important qualitative DQI for anal ytical assessment and is critical when considering the combination of datasets with the same anal yt es . The assessment of this DQI determines if analy

tical results reported are equivalent to data obtained from similar analy ses . Only comparable datasets can be readily combined and used for subsequent statistical evaluations
. For example, if two separate investigations of the same site utilized different anal ytical methods with different sensitivities, combining the data may violate the underl yi ng assumptions of whic h the statistical tests depend
. The potential differences in precision and bias could be misinterpreted as within
- unit sampling error. As with any

decision about the usability of data, it is important to consider the decision that the data are meant to support . Separate determinations of the comparability of datasets may

be necessary for each decision the data are used to support
. Examples in the uses of comparability are found in Chapter 3, Section 3.4.1.QAPP Handbook B - 10 December 2012 Agency Review Draft **B.8 COMPLETENESS**  Completeness is a measure of th e amount of usable data obtained from a measurement sy stem, expressed as a percentage of the number of measurements that should have been collected according to the study design (i.e., measurements that were planned to be collected)
. Percent completeness i s calculated using the formula: Percent Completeness = (number of usable measurements) x 100 % (total number of measurements planned) Completeness is one measure of how well a sampling and anal ys is design was implemented
. A data set that is 100% complete is the result of careful planning and precise implementation of the data collection plan
. Completeness is not intended to be a measure of representativeness; for example, it does not describe how closel y the measured results reflect the actual con centration or distribution of the pollutant in the media sampled, but it may be a contributing factor
. A complete data set may

or may not achieve the project objective depending on how well the sampling plan reflects the conceptual model , how accurate the conceptual model was prior to sampling, and whether the distribution of the reporte d data are similar to th ose anticipated
. When considering completeness, attention should be given not only to simply complete samples but also to the measurements on anal yt es within a sample. If, for example, m easurements on 6 anal yt es w ere requested , but only 4 actually

made, this sample would be classified as “incomplete” if the missing anal yt e measurements were part of the MQOs or DQOs. It follows that for some instances, completeness should be defined in terms of anal yt e/sample combinations rather than samples. For example, suppose measurements of six analytes were requested in 20 samples, but in one sample only four were actually

performed or found acceptable readings; the entire sample would be found “incomplete” with a completeness of 19/20 = 95%. If completeness is defined in terms of anal ys t/sample combinations, then the completeness would be 118/120 = 98.33%. The important question for decision makers is whether the number of measurements is sufficient to support the decision to be made
. For example, there could be only 70% data completeness (30% lost or found invalid), but, because of the nature of the study design, the results could still be representative of the t arget population and yi eld valid estimates
. Conversely , a data set with much higher completeness but systematicall y omitted or rejected data may be insufficient to yi eld valid estimates of the parameters of interest
. Key questions for the DQI of completene ss would include:
 W as the number of field samples taken for each matrix and analy tical group the same as that documented in the sampling plan?  Were all the associated quality control samples taken?  Were all the samples delivered to the lab anal yz ed?  Were the samples analyzed with the appropriate method as documented in the QAPP? and  Were any of the sample results rejected due to quality concerns?QAPP Handbook B - 11 December 2012 Agency Review Draft It is suggested that the QAPP set a quantitative goal for completeness
. If that goal is achieved, then no further consideration of this DQIis necessary
. If the completeness goal is not achieved, the QAPP should make clear how the project team will proceed and what options might exist. It is important that the completeness goal is potentially attainable (95% when using onl y 10 samples, for example) and that it is linked to the project’s objectives. Itis important to distinguish between incompleteness of investigative samples and QC samples, as the loss of the latter can be of less importance than the former. **B .9**  **SENSITIVITY** Sensitivity

generall y refers to the capability of a method or instrument to discriminate between small differences in analyte concentration
. Both the precision of the instrument and the slope of the calibration curve limit sensitivity
. Chemi sts typically define sensitivity

as the slope of the calibration curve at the concentration of interest (Skoog 1985 , Currie 1995). If two methods have equal precision, the one having a steeper calibration curve will be the more sensitive
. Sensitivity can also be evaluated from the standard deviation of replicate anal ys es at any concentration level or can be evaluated from the confidence bound on a calibration curve
. It represents the minimum difference in two samples that can be distinguished with a defined confidence (Taylor, 1987). The sensitivity

indicators of primary

interest relate to limits of detection
. The detection limit (DL) is generall y considered to be the minimum true concentration of an analyte producing a non - zero signal that can be distingui shed from the signals generated when no concentration of the anal yt e is present, with an adequate degree of certainty
. There are a plethora of different approaches to the determination of detection limits that can be broadl y categorized into two groups: th ose that evaluate the statistical variability of instrument responses with and without the anal yt e present . There are several critical aspects of determining detection limits that have historically

been overlooked:
 the variability

of instrument responses are commonly heterogeneous or heteroscedastic (i.e. standard deviation is not linear over the concentration range) ;  most common methods only consider false positives (ty pe I or α error) at the Detection Limit ( DL ) , ignoring the importance of false negatives (ty pe II or β error) ( The false negative rate at the DL is 50%
. Therefore, there is a 50% chance of error when reporting a sample result as <DL (or <MDL) when nothing is detected);  the distribution of measured values around the true detection limit is as sumed to be normally distributed
. ( In reality

that assumption at the true DL is likely false
. Unfortunatel y we cannot verify that assumption because we do not have statistical confidence at the DL );  instrument responses, variability, and subsequent detection limits are greatly affected b y the sample matrix, instrument conditions, and even the anal ys t ; and  many anal ytical methods produce non zero signals even when a target anal yt e is not present . The Limit of Detection (LOD) is the smallest concentration o f anal yt e present in a sample in order to be detected at the 99% confidence level
. The LOD differs from the DL in that the probability

of a false negative is 1%
. The LOD is usually >1 to 2 times the DL
.QAPP Handbook B - 12 December 2012 Agency Review Draft The quantitation limit (QL) is another concept that is very important for environmental decision making
. The QL relates to the lowest concentration at which the method is expected to be able to quantify the amount of analy

te present in the sample. The QL is usuall y 5 to 10 times the DL
. The Limit of Quantit ation (LOQ) is the smallest concentration with a specified level of precision and bias. That is, samples with concentrations at or above the LOQ can be quantified with a known level of accuracy . Detection and quantitation levels (DLs, QLs, LODs, and LOQs) vary by anal yt e and b y matrix and often vary among laboratories. It is critical that when referring to these levels the specifications are made very clear. Analy tical capabilities are constantly improving, resulting in greater sensitivity and lower detection limits
. This improvement in anal ytical capabilities is frequentl y the vehicle that drives regulatory

and, hence, project requirements
. Investigators often base their sensitivity requirements upon anal yt ical method capabilities rather than upon project
- specific objectives
. The problem for a project team is to determine the levels of sensitivity that will generate data adequate for decision making, to establish MQOs based on this evaluation, and to be sure that the indicator of sensitivity

used to evaluat e a particular method appropriately reflects the performance of the method in the particular matrix of interest
. MQOs tie the required measurement quality

to the project DQOs
. The project team should alway s consider the needed sensitivity of a measurement prior to requesting laboratory

analy ses . Once the needed sensitivity is determined, the project team can work with laboratory

personnel to choose the appropriate anal ytical method
. Sensitivity is dependent on the sample matrix, the analy tical method, the instrument used, and the operator conducting the anal ys is
. The sensitivity of the analy ses potentiall y impacts data usability for a given decision
. A couple of issues that are frequently encountered in environmental projects relating to sensitivity

are des cribed below, along with suggestions for handling them. Project action levels are numerical values used b y the decision
- makers as a bright
- line for selecting one of two or more alternative actions (e.g. remediate or leave in place)
. Action levels may be regulatory

thresholds such as a M aximum Contaminant L evel (MCL) , a risk
- based or toxicity reference benchmark, or a treatment standard
. Some of these action levels may be at concentrations well below the DL of many anal yt es . However, because of the great un certainty at the DL , the QL should alway s be less than project action levels
. During project planning, project teams may

be faced with the dilemma of targeting action limits at or below QL s . It is recommended that the laboratory p erform project
- specific c onfirmation of action limits in each matrix of concern
. Some ty pes of DL s such as MDL s are determined by the laboratory under ideal conditions using a clean reference matrix
. Analy tical sensitivity is impacted by the presence of interferences
. Therefore, the QAPP may include a requirement to perform a sensitivity

verification study

for each matrix
. The laboratory can verify

if the project action levels can be met by

spiking the appropriate matrix at the action level for each analy te . The verification proced

ure for establishing the ability to meet the action levels must empirically demonstrate quantifiable estimates of precision and bias at the action level
. When precision and bias do not meet project requirements at project action levels, several options should be considered to ensure that the QL s are below the project action levels:QAPP Handbook B - 13 December 2012 Agency Review Draft 1. Modification of the proposed method
. In some cases, it may

be possible to modify

the proposed method to achieve improved sensitivity
. Possible modifications that could have a beneficial impact on sensitivity include:
 extracting more sample volume or mass, or decreasing the final extract volume (bearing in mind that problems in homogeneity and interferences may result in an increase in bias) ;  including additional cleanup steps (sample preparation steps to eliminate interferences); or  use a different wavelength or ion for quantitation. Performing anal ys es in selected ion monitoring mode rather than full
- scan mode during mass spectrometry

anal ysis is an example of a modification to the method that may improve sensitivity
. 2. Use a different method with better sensitivity
. For many environmental analy sis ne eds (e.g., measuring the concentration of pol yc yclic aromatic hydrocarbons in sediment), multiple analy sis methods are available
. The differences between methods may include cost, availability , and sensitivity
. If the project action level is below or near the QL for a particular method, it may be possible to identify and implement a different method for anal ys is for which the QL is well below the project action level
. For example, if running inorganic anal ys es using Inductively Coupled Plasma Optical Emissi on Spectroscop y ( IC P - OES ), is not achieving the desire level of sensitivity, switching to IC P - M ass Spectroscop y (ICP
- M S ) anal ys es might provide the additional sensitivity to meet objectives. 3. Consider alternative action limits when it is reasonable to ques tion the applicability of a pre -set action level for a particular project
. For example, taking action at the level of the MCLif there is no potential for the water to be studied (e.g., water in an ephemeral stream) to be used as a drinking water source may be overl y conservative, and other, more project
- specific action limits could be considered. 4. Report all analytic results with their associated uncertainties (precision and bias)
. Although the uncertainty of results below Q L s is relatively large, as long as the uncertainty is known a determination can be made regarding the usability of the data in relation to the project action level. If all results are reported with their associated uncertainties, the project team will be abl e to ascertain the level of confidence that would be achieved if they

make a decision regarding an action limit that is below the QL . One way to establish the uncertainty (precision and bias) in the estimated results is using a technique developed b y Hubaux and Vos ( 1970)
. The Hubaux
- Vos method is an extrapolation method that utilizes the linear regression of the calibration curve
. The DL is determined b y the width of the confidence bands of the calibration process
. This approach has the added benefit of co nsidering both false positive and false negative instrument readings
. If this approach is utilized, the detection limits should be reported along with the total number of calibration points, the number of replicates, and the associated error rates used to define the precision and bias bounds.QAPP Handbook B - 14 December 2012 Agency Review Draft Another issue relating to sensitivity is the way

that very low results are reported as there is a great deal of uncertainty associated with very low results
. The high uncertainty makes it difficult for chemists to be able to state what concentration of an anal yt e is present in the sample
. One way that chemists handle this is by reporting that the result is less than some detectable or quantifiable amount
. There are several different conventions for how this is done suc h as:
 c hoosing protection against false positive results (incorrectly reporting that an anal yt e is present in a sample when in reality it is not present) such as: o results below the MDL are reported as undetectable ( < DL) , o results between the MDL and the practical quantitation limit (PQL) ar e reported as estimated) , and o results above the PQL are reported directl y .  Choosing protection against false negative results (incorrectly reporting that an anal yt e is not present in a sample when it actually is present) such as: o all results below the laboratory DL are reported as undetectable (as <LOD, where LOD is one or more times the DL) , o results between the DL an d L OQ are reported as estimated , and o results above the LOQ are reported directl y . With multiple reporti ng conventions in use, many different methods for setting the DL ( EPA 2010 ) , and many different methods for establishing the QL , it is clear that Project Managers need to document in the QAPP which convention they will adopt
. If the standard practices of the laboratory are deemed acceptable for the project, it should be stated in the QAPP and the laboratory

SOPs relating to their DL s, QLs , and reporting of results should be summarized and/or attached
. If the DL is going to be c alculated using a matrix
- specific method (e.g., Hubaux and Vos 1970 ), then the methods should be well documented in the QAPP
. Finally , while setting a sensitivity level that must be reached to attain DQOs would appear to be straightforward, it may

not be
. Is it the DL or the QL , or both that need to be defined in MQOs? Is a reporting convention that protects only against false positives acceptable for the needs of the project? Does the project call for a reporting convention that protects against false neg atives? In the arena of DLs , Q L s and reporting conventions for data at very low levels, there is not an overall best way

to proceed as there are just a few common practices that have become relatively standard for environmental data analy s e s . If DQOs indic ate that these issues might be important (i.e., the concentration levels at which decisions must be made are very low), it is suggested that the project team carefully consider how MQOs for sensitivity are defined, and how the laboratory

statement of work is written to ensure that the MQOs can be achieved. **B.10 THE ROLE OF DQI**  **s IN THE**  **PROJECT LIFECYCLE**  DQIs play

a number of important roles during the project planning phase such as calculating relevant indicators from historical data to support the design o f new efforts, establishing MQOs , identifying future DQI needs, and specifying quantitative and qualitative requirements
. The role of DQIs during project planning focuses on assumptions (ty pically based on analy sis of historical performance data) made during the development of DQOs and an associated statisticalQAPP Handbook B - 15 December 2012 Agency Review Draft design of a project
. For example, design optimization may involve an anal ysis of the major sources of sampling and measurement errors, as well as spatial and temporal variability that will contribut e to uncertainty
. Total study error impacts the ability

to achieve the limits on decision error specified during Step 6 of the DQO process ( EPA 2006a)
. Depending on the relative contribution of different components of total study error (especially components of total study precision), different choices may be made to cost
- effectivel y achieve the specified DQOs
. Project DQOs together with individual MQOs for precision, bias, and sensitivity are critical inputs to developing alternative sampling and analysis designs
. Alternative designs should be developed that reflect different combinations of statistical sampling and measurement schemes
. The expected performance of these designs may be c alculated using these indicator DQOs/MQOs as inputs ; the output of this effort is only as valid as the assumptions (including the quality and relevance of the indicators) that are made
. In addition, once the critical DQIs are identified (those to which the final data quality is expected to be most sensitive), important design choices concerning QA/QC samples that enable estimates of the DQI s can be made
. The purpose of such an anal ysis is to determine the likely output before the actual obligation of resources. Understanding the critical relationships between DQOs and MQOs is essential to the generation of data of known quality
. Decisions about what DQI s are appropriate in a study and what MQOs are to be achieved to meet the project DQOs should be documented in the QAPP . By conducting interim evaluati ons of the performance of the measurement sy stem, project staff can identify

whether to implement adjustments or corrective action to keep measurement s ys tems in control before the project budget and schedule are expended
. This is especiall y true for ongoing measurement programs su ch as monitoring efforts required for compliance programs, because mid
- course corrections can be made based on feedback from the last sampling effort, or last series of samples
. To support these QC measures, specific DQIs should be identified and data coll ected to support their calculation
. In addition, MQOs assess and interpret the QC data results
. Based on these types of comparisons, corrective actions can be taken as needed to improve project performance and increase the probability that data will be ade quate to support the intended use
. Too often, QC data are generated, but not evaluated at all, or not evaluated in a timely , meaningful way
. The DQIs discussed in this document can be used not only in a real
- time QC mode, but also in the assessment of the overall performance of a program over some period of time such as a field season
. For example, indicators of precision, accuracy , and sensitivity can be evaluated over the field season to determine how well the program was able to measure critical variabl es . Following a data collection effort, determination of the adequacy of the data can be made via data verification, validation , and an evaluation of usability
. The purpose of those data review steps is to determine whether the data set will support a dec ision with the desired degree of certainty . It is important to consider the performance and representativeness of the measurement effort prior to reaching conclusions regarding data adequacy
. Step B of data validation (see Appendix E) focuses specificall y on attainment of the MQOs set as quantitative goals for DQIs
. Data usability

then revolve s around whether an adequate number of samples was obtained, given the observed measurement, spatial and temporal variability , and given the actual magnitude ofQAPP Handbook B - 16 December 2012 Agency Review Draft the me asurements made (relative to levels of concern) to meet the DQOs
. If a data collection effort fails to generate adequate data to meet DQOs , then interest in DQIs is heightened, especiall y if there is a desire to “diagnose” which assumptions proved valid or invalid
. Having documented the basis for the design based on DQIs, project staff can identify where the s ys tem failed to perform adequately and determine what additional data will be required to support the decision
.QAPP Handbook C - 1 December 2012 Agency Review Draft **APPENDIX C QUALITY CONTROL SAMPLES**  Figures C - 1 and C
- 2 depict a measurement process that involves several steps
. Each step has the potential to introduce variability or bias that might influence the quality of project data
. Estimates of the components of overall study variability and the relative contribution of measurement (within
- unit) precision are important inputs to the statistical design process
. The QAPP should describe the processes used to estimate and monitor the magnitude of at least the most important potential measurement error sources
. Q uality planning for measurements simultaneously

establishes MQOs appropriate for the project and data use and defines the required DQ Is . The data sources for DQIs are often derived fro m samples inserted in the sampling process by

field or laboratory

personnel on a frequency specified in the QAPP . Figures C - 1 and C
- 2 suggest the types of QC samples that might be identified as the source of the underlying data for precision ( Figure C - 1 ) and bias ( Figure C - 2 ) DQI c alculations.

**Figure C**  **- 1**  **. Total Sampling and Measurement Process Denoting the Use of QC Samples** 

**to Measure Components of Total Study Precision** QAPP Handbook C - 2 December 2012 Agency Review Draft **Figure C**  **- 2**  **. Total Sampling and Measurement Process Denoting the Use of QC Samples** 

**to Measure Components of Total Study Bias**  Figures C
- 1 and C
- 2 are illustrative only with “statistical sampling” error (variance calculated from the sample analy

tical results) far exceedi ng all other errors (variance calculated from the QC sample anal ytical results). The results from routine QC samples often can be used to construct useful DQIs; however, some thought should be given to what support such data provide to the quality

goals se t out in the QAPP . One use of routine QC results that is generally applicable is assessment of the laboratory's internal operations; for example, the routine analy sis of calibration blanks to test for inorganic contamination or a problem in s ys tem calibration
. In this case, a DQI could be just the actual blank result, expressed in appropriate units, with an MQO equal to the laboratory ’s internal acceptance criterion
. A poor result from a routine calibration blank should result in an immediate corrective action by the laboratory
. This action may be limited to simply

rerunning a calibration blank to demonstrate that the first result was an aberration
. While it is important to verify the laboratory

is monitoring and controlling the quality of its internal operations, results from samples like a calibration blank are less important in project planning from the DQO perspective than other potential sources of error that may not be controlled
.QAPP Handbook C - 3 December 2012 Agency Review Draft The same can be said for overall anal ytical calibrations
. A properl y calibrated anal ytical sy stem is alway s required and expected for project samples analy s e s . Calibration data verify internal quality control, which is assumed to take place and is therefore not directl y discussed during the DQO planning process
. Instead, the qu ality attributes of greatest utility from a DQO planning and statistical design perspective include estimates of the overall (total) study variability

and an understanding of the relative contribution of significant components of this total
. The goal of an al yz ing components of bias is to reduce the most manageable components to minimal levels so the Total Study

Error (as measured b y the total sample variability) is reduced to a combination of random errors (the components of variability). T o establish the M QOs, the sensitivity of overall study

error to the precision of various subcomponents of the total (e.g., the anal ytical component of measurement error) should be examined
. A simplistic approach for examining the relative contribution of different components of total study

variability

begins from the assumption that total variability can be separated into individual, additive, components:



### = + + + ⋯ Using the framework for decomposition of errors shown in Figures C
- 1 and C
- 2 the sum of errors i n an environmental investigation might be



### = + = + + where: T = total study error w = within
- sampling unit error b = between
- sampling unit error s = small
- scale error m = measurement error By identifying the largest of these components, decisions on method selection and laboratory requirements (e.g. repeated measurements) can be made. Working with variances can be difficult conceptuall y because the terms are squared
. A visualization tool usi ng the more intuitive standard deviation statistic may be used for examining the additive relationship between precision components
. The visualization t ool takes advantage of Py thagoras' Theore m concerning the sides of a right triangle (the square of the hyp otenuse is equal to the sum of the squares of the other two sides)
. Figure C - 3 is a graphical representation of the most basic division of total study

variability

into the within
- unit (measurement and small
- scale) and between
- unit (field or spatial) vari ability . Note that in this model the lengths of the sides of the triangle are directl y proportional to the standard deviation, or precision, of the different components . The total standard deviation is obtained from the results of the actual randomly locat ed field samples while the estimate of within
- unit precision is best estimated using collocated samples
. The between
- unit variability is inferred from the other two sides of the triangle
. The dimensions in Figure C - 3 represent the relative scales often enc ountered in environmental sampling, that is, between
- unit variability generall y dominates total study variability
.QAPP Handbook C - 4 December 2012 Agency Review Draft The requirements for precision can be stated at the very highest level (e.g., the total study error must not be greater than x%), or separate d into between
- and within
- unit precision, or even further as suggested b y Figure C - 3 . The QAPP should document quantitative MQOs for both between
- unit and within
- unit precision, and perhaps for more specific sources of imprecision that is needed to meet project
- specific needs
. In addition, the QAPP should document the corrective actions to be performed in the event any of the performance criteria are not met
.

**Figure C**  **- 3**  **. Components of Total Study**  The QAPP should i dentify QC activities needed for each sampling, anal ys is, or measurement technique
. For each required QC activity, the associated method or procedure, acceptance criteria, and corrective action should be listed
. As standard methods are often vague or incomplete in specifying QC requirements, s imply relying on the cited method to provide this information is usually

insufficient
. QC activities for the field and the laboratory include, but are not limited to, the use of blanks, duplicates, matrix spikes, laboratory control samples, surrogates, or second column confirmation
. A list of various kinds of QC samples, as shown in Figures C - 1 and C
- 2 , are shown in Table C - 1 . **Table C**  **- 1**  **. Uses of Various**  **QC Samples QC Sample**  **Description Purpose**  **Field or**  **Lab Bias or**  **Precision Field**  **duplicate, collocated**  **samples** T wo or more independent samples

collected from side
- by
- side locations at the same point in time and space T o assess precision of the total method, including sampling, analysis, and site heterogeneity **Field Precision** 

**Field duplicate**  **(split), subsamples**  D uplicate samples resulting from

one sample collection at one sample location (may be repeated collection or split of original sample) T o evaluate the effects of within
sample heterogeneity **Field Precision** QAPP Handbook C - 5 December 2012 Agency Review Draft **QC Sample Description**  **Purpose Field**  **or Lab**  **Bias or Precision**  **Matrix spike duplicate**  A duplicate sam ple prepared simultaneously as a split of the matrix spike sample, each spiked with identical, known concentrations of targeted analytes T o determine the precision of the laboratory analytical process for specific analytes in a sample matrix **Field Precisio**  **n**
**Equipment blank**  A clean water sample poured over

or through decontaminated field sampling equipment T o assess the adequacy of the decontamination process (also called rinse blank or rinsate blank) **Field** Bias
**Field blank** A clean sample exposed to

sampling conditions, transported to the laboratory, and treated as an environmental sample U sed to provide information about contaminants that may be introduced during sample collection, storage, and transport (may also serve as a rinsate blank) **Field** Bias

**Field matrix spike**  A sample prepared by adding a

known concentration of a target analyte to an aliquot of a specific homogenized environmental sample for which an independent estimate of the target analyte concentration is available A ccom panied by an independent analysis of the unspiked aliquot of the environmental sample, spiked samples are used to determine the potential bias introduced due to specific matrix effects **Field** Bias

**Trip blank** A clean water sample transported

from the sampling site to the laboratory for analysis without having been exposed to sampling procedures T o assess whether contamination is introduced during sample shipment (ty pically analyzed for volatile constituents) **Field** Bias

**Instrument replicate**  T wo or mo re analyses of the same

sample designed to evaluate the precision of the analyses (also known as analytical replicates) T o evaluate the precision of the analyses at the instrument level only Lab **Precision**
**Laboratory duplicates/**  **replicates** T wo or more representative portions

taken from one homogeneous sample by the laboratory and analyzed in the same laboratory T o assess the overall laboratory preparatory and analytical precision Lab **Precision**
**Laboratory split sample**  T wo or more representative portions

of the same sample, analyzed by at least two different laboratories and/or methods. Prior to splitting, a sample is mixed (except volatiles, oil and grease, or when otherwise directed) to minimize sample heterogeneity T o assess precision, variability, and data comparability between different laboratories Lab **Precision**QAPP Handbook C - 6 December 2012 Agency Review Draft **QC Sample Description**  **Purpose Field**  **or Lab**  **Bias or Precision**  **Instrument performance**  **check sample** A sample of known composition

analyzed concurrently with environmental samples to verify the performance of one or more components of the analytical measurement process (e.g., retention time, resolution, recovery, degradation, and calibration verification) T o evaluate the potential bias introduced from the instrument (as part of the measurement) Lab Bias

**Laboratory fortified blank**  A low
- level laboratory control

sample (e.g., at the quantitation limit) (also known as laboratory spike or blank spike) T o evaluate laboratory preparatory and analytical sensitivity and bias for specific compounds Lab Bias
**Matrix spike** A known concentration is injected

into the matrix and then analyzed To determine the recovery rate and therefore the bias Lab Bias **Method blank (or extraction**  **blank)** A sample of a matrix similar to the

batch of associated samples in which no target analytes or interferences are present at concentrations that impact the analytical results, processed and analyzed simultaneously and under the same conditions as the samples T o evaluate the potential overall positive bias introduced by sample processing and measurement Lab Bias

**PT sample** A sample of composition unknown

to the laboratory or analyst, but known precisely by the PT sample vendor (also known as PE samples) T o assess potential bias and therefore capability to produce results within acceptable criteria PT samples can fall into three categories: (1) prequalification, conducted prior to a laboratory beginning project work, to establish initial proficiency; (2) pe riodic (e.g., quarterly, monthly, or episodic), to establish ongoing laboratory proficiency; and (3) batch
specific, which is conducted simultaneously with analysis of a sample batch. Lab Bias

**Reagent blank** An aliquot of clean water or solvent

analyzed with the analytical batch and containing all the reagents in the same volume as used in the processing of the sample T o evaluate the potential bias introduced through reagents used during testing Lab Bias
Note that although field
- level QC will provide val uable information on total error it cannot identify the source of that error. It should be expected that the contributions to total error from the laboratory

QC samples will be quite small but are useful in identifying where relativel y large amounts of error are located. Within the QAPP, appropriate QC activities should be identified (Figures C
- 1 and C
- 2 may be helpful in thinking about what ty pes of QC activities fit project needs)
.QAPP Handbook C - 7 December 2012 Agency Review Draft The QAPP should state the frequency of anal ys is for each type of QC activity , and the spike compounds , sources , and levels
. The QAPP should also state or reference the required control limits for each QC activity , and the corrective action required when control limits are exceeded , and how the effectiveness of the correcti ve action shall be determined and documented. The QAPP should d escribe , reference , or attach the procedures to be used to calculate applicable statistics (e.g., pr ecision and bias)
. Copies of the formula e are acceptable as long as the accompanyi ng narrative or explanation specifies clearl y how the calculations will address potentially

difficult situations such as missing data values, “less than” or “greater than” values, and other common data qualifiers.QAPP Handbook D - 1 December 2012 Agency Review Draft **APPENDIX D DATA REVIEW, VERIFICATION,**  **AND VALIDATION FOR PROJECT PURPOSES**  Data review is the process in which data are examined and evaluated to varying levels of detail and specificity b y a variety of personnel who have different responsibilities within the data
management process. The process includes verification, validation , and usability assessment. These QAPP element s encompass the data review activities used to ensure that only scientifically valid data that are of known and documented quality and meet the DQOs ( PQOs depending on the organization) are used in making environmental decisions. The approach used for data review of a project must be appropriate to project requirements. Although data revi ew takes place after the data have been generated, determination of the ty pe of data review that is required to meet DQOs begins during the planning phase of the project. Key questions regarding data review that must be answered during the project planning stage include (but are not limited to ) :
 What M easurement Q uality Objectives (MQOs) are necessary to achieve the required DQOs such as precision, accuracy , representativeness, comparability, sensitivity, and completeness? ( s ee also EPA 2006a)  What data rev iew inputs, activities, and outputs will be required for this project?  What entities will be responsible for each step of the data review process and what are their relationships to those responsible for the data
- production process?  How will the implementation of the data review process and its results integrate with the overall project decision timeline? and  What is the extent of data review and the availability and appropriate use of streamlining tools? There are three distinct data review steps that are used to ensure that project data quality needs are met. These data review steps are required for all data collected and used in environmental p rojects
. Data verification entails c onfirmation b y examination and provision of objective evidence that the valid ated information fulfils specified requirements (sampling and analy tical) , or requirements mandated by a contract or agreement . Data validation means c onfirmation by examination and provision of objective evidence that the particular requirement for which the data or information was collected is fulfilled. It includes the process of checking whether the information meets the project’s specifications. Verification and validation extends to evaluating against criteria based on the quality objectives developed in the QAPP (e.g., the QAPP and MQOs )
. The purpose of validation is to assess the performance of the sampling and anal ys is processes to determine the quality of the specified data
. It is divided into two subparts:
 assess and document compliance w ith meth ods, procedures, and contracts; andQAPP Handbook D - 2 December 2012 Agency Review Draft  assess and document an evaluation of the ability of the data to meet the M QOs in the Q APP. Table D - 1 describes the objectives, scope, steps, and output of data review associated with each process term. The table identifies where the scope of the terms used or the steps involved in the process are expansions of current practice. **Table D**  **- 1**  **. Three Steps of the Data Review Process Step**  **Objective Scope**  **Activity** Verification Review to see if all the

expected data are present Sampling Analysis Completeness check Validation Assess and document the

performance of the field
sample collection process Assess and document the performance of the analytical process Sampling Analysis Step A. Check compliance with

method, procedure, and contract requirements Step B. Compare with M QOs from the QAPP Usability Evaluation Assess and document

usability to meet PQOs Sampling Analysis Evaluate usability of data against

DQO s and the decision to be made Each step of the process is critical to the overall assessment of data quality and each step builds on the outcome of the previous step. The level of data review (types and amount of data reviewed) should be appropriate to the PQOs. Streamlining data review (validation in particul ar) is an option to consider that can potentiall y b ypass some validation requirements, if allowed b y the project’s data quality

needs. T o perform the data review steps described above, reported anal ytical data must be supported by complete data packages, a s defined in the QAPP
. Data packages include sample receipt and tracking information, chain
- of
- custody records, tabulated data summary forms, and raw anal ytical data for all field samples, standards, QC samples, and all other project
- specific documents that are generated
. If relevant raw data or sample information are not available or adequate to document data quality , then data review cannot be performed, and re - sampling or re
anal ys is must be considered . Existing data and model inputs/outputs should also be evaluated during data review
. This appendix describes what data review information should be included in the QAPP for each of the three data review steps: verification, validation ( S teps A and B ), and evaluation of usability . Table D - 2 is provided as an example of inputs for data review and identifies the dat a review process step to which each input applies
. These are onl y examples and are not intended to be either a minimum or comprehensive list of inputs. Items 1 through 12 are to be found in the planning documents, 13 through 35 in the anal ytical data requirements document, 36 through 55 in the sampling documents.QAPP Handbook D - 3 December 2012 Agency Review Draft **Table D**  **- 2.**  **Example Inputs to the**  **Data Review Process**

**Item Verifi**  **cation Validation**  **Step A Comp**  **li ance**  **Validation**

**Step B Comparison**  **1** Evidence of required approval of plan (QAPP) **X 2**  Id entification of personnel (those involved in the project and those conducting verification steps) **X 3**  Laboratory

name **X 4**  Methods (sampling and analy sis) **X X**  **5** Performance requirements (including QC criteria) for all inputs **X X**  **X 6**  Project quality

objectives **X X**  **7** Reporting forms **X X**  **8** Sampling plans, location, maps, latitude, longitude, grids, and sample ID numbers **X X**  **9** Site identification **X 10**  SOPs (sampling and analytical) **X X**  **11** Staff training and certification **X 12**  List of project
- specific analy tes **X X**  **13** Case narrative **X X**  **X 14**  Internal laboratory

chain of custody **X X**  **15** Sample condition upon receipt and storage records **X X**  **16** Sample chronology

(time of receipt, extraction, and anal ys is) **X X**  **17** Id entification of QC samples (sampling or lab, temporal, and spatial) **X X**  **18** Associated (batch or periodic) PT sample results **X X**  **X 19**  Communication logs **X X**  **20** Copies of laboratory

notebook, records, prep sheets **X X**  **21** Corrective action reports **X X**  **22** Definitions of laboratory qualifier flags **X X**  **X 23**  Documentation of corrective action results **X X**  **X 24**  Documentation of individual QC results (e.g., spike or duplicate) **X X**  **X 25**  Documentation of laboratory method deviations **X X**  **X 26**  Electronic data deliverables **X X**  **27** Instrument calibration reports **X X**  **X 28**  Laboratory

name **X X**  **29** Laboratory

sample identification numbers **X X** QAPP Handbook D - 4 December 2012 Agency Review Draft **Item Verifi**  **cation Validation**  **Step A Comp**  **li ance**  **Validation**

**Step B Comparison**  **30** QC sample raw data **X X**  **X 31**  QC summary

report **X X**  **X 32**  Raw data **X X**  **X 33**  Reporting forms, completed with actual results **X X**  **X 34**  Signatures for laboratory sign
- off (e.g., laboratory

QA manager) **X X**  **35** Standards traceability records (to trace standard source from National Institute of Standards and Technology, for example) **X X**  **X 36**  Chain of custody **X X**  **37** Communication logs **X X**  **38** Corrective action reports **X X**  **X 39**  Documentation of corrective action results **X X**  **X 40**  Documentation of deviation from methods **X X**  **X 41**  Documentation of internal QA review **X X**  **X 42**  Electronic data deliverables **X X**  **43** Id entification of QC samples **X X**  **X 44**  Meteorological data from field (e.g., wind or temperature) **X X**  **X 45**  Sampling instrument decontamination records **X X**  **46** Sampling instrument calibration logs **X X**  **47** Sampling location and plan **X X**  **48** Sampling notes and drilling logs **X X**  **X 49**  Sampling report (from field team to project manager describing sampling activities) **X X**  **X 50**  External assessment report **X X**  **X 51**  External PT sample results **X X**  **52** Laboratory

assessment **X X**  **53** Laboratory

QA plan **X X**  **54** Detection limit study

information **X X**  **X 55**  Lab accreditations **X X**  **D .1**  **VERIFICATION** Verification is a completeness check that is performed before the data
- review process continues in order to determine whether the required information (the complete data package) is available for further review
. It involves a review of all data inputs to ensure that they are present . Table D - 2 provides examples of the inputs for conducting the completeness check.QAPP Handbook D - 5 December 2012 Agency Review Draft Although this step is not designed for use in qualitative review (e.g., a compliance chec k that takes place during step B of the validation process), it is essential for ensuring the availability of sufficient inform ation for subsequent steps of the data review process. The planning process should establish verification procedures, which should be documented in the QAPP to ensure that data are evaluated properly , completel y, and consistently for use in meeting DQOs. The procedures should address the following:
 the process that will be used to verify sample collection, handling, field analy sis, and anal ytical laboratory

project data; and  the procedures and criteria that will be used to verify data information operation s. ( These operations include, but are not limited to, the electronic and/or manual transfer, entry, use, and reporting of data for computer models, algorithms, and databases; correlation studies between variables; and data plotting .) Verification inputs may

include items such as those listed in Table D - 2 . The description should detail how each item will be verified, when the activity will occur, and what documentation is necessary . Internal or external is in relation to the data generator. The resulting tables will describe the following:
 how sample collection, handling, and anal ys is procedures will be verified;  how verification of field sampling, handling, and anal ys is activities will be documented (e.g., QC signatures in field logs an d QC checklist s );  which sampling, handling, on
- site anal ytical, and off
- site laboratory

data will be verified internally at the data
- generator level;  the end product of laboratory verification (e.g., laboratory
- qualified data); and  which sampling, on
- site anal ytical, and off
- site laboratory

data will be verified by

entities external to the data generator. **D .2**  **VALIDATION** The QAPP planning process must establish validation procedures and criteria
. Project
- specific validation procedures are developed to identify and qualify

data that do not meet measurement performance criteria
. Validation procedures and cr iteria are documented in the QAPP to ensure that data are evaluated properly , completel y, and consistently for use in meeting DQOs
. Validation guidance and documents may be attached to or referenced in the QAPP
. EPA has issued Guidance for Labeling Externally Validated Laboratory Analytical Data for Superfund Use , OSWER No. 9200.1 - 85, EPA 540 - R - 08 - 005 ( EPA 2009 c ) , which provides a standardized system for publishing the outputs of data validation
. This or any other reasonable methodology

for validation may

be used provided it is properl y documented
.QAPP Handbook D - 6 December 2012 Agency Review Draft The discussion of validation in the QAPP should address the following: the process that will be used to validate sample collection, handling, field anal ys is, and anal ytical laboratory project data;  the specific validation process that will be used for each anal ytical group, matrix, and concentration level; and  the procedures and criteria used to validate data information operations .(These may include, but are not limited to, the electronic or manual transfer, entry , use, and reporting of data for computer models, algorithms, and databases; correlation studies between variables; and data plotting .) Validation inputs include items such as those listed in Table D - 2. The description should detail how each item will be validated, when the activity will occur, and what documentation is necessary . The resulting tables will describe the following:
 how sample collection, handling, and anal ys is procedures will be validated against the measurement performance criteria;  how validation of field sampling, handling, and analy sis activities will be documented (e.g., QC signatures in field logs and QC checklist s );  which sampling , on - site anal ytical, and off
- site laboratory

data will be validated;  the evaluative procedures used in validation to assess overall measurement error associated with the project, including DQIs; and  the individual, identified by title and organizational a ffiliation, who is ultimately responsible for data validation. This is the person (lead chemist or project chemist) who will sign the project validation reports. In addition, the QAPP should identify the matrices, analy

tical groups, and concentration level s that each entity

performing validation will be responsible for, as well as the criteria that will be used to validate those data. **D .2.1**  **Validation Activities**
**– Step A**  **: Compliance**  The examples listed in Table D - 3 are of specific activities that may occur during an environmental project under S tep A of the validation process (compliance with methods, procedures, and contracts) for both sampling and analy tical data. Although these activities are organized separatel y, they may be performed at the same time and/or b y the same people as verification and S tep B of validation activities. **D .2.2**  **Validation Activities**
**– Step B**  **: Comparison**  The examples listed in Table D - 4 are of specific activities that may occur during an environmental project under Step B of the validation process (comparison with MPCs documented in the QAPP) for both sampling and analy tical data
. These activities require that the validators have a complete cop y of the QAPP, and they often involve all or parts of the project team . Some of the activities listed for Step A have a QAPP
- specific review element and are therefore also listed as activities under Step B .QAPP Handbook D - 7 December 2012 Agency Review Draft **Table D**  **- 3**  **. Validation**  **Step A :**  **Compliance with Met hods, Procedures, and Contracts**  **Activity Data Deliverables and**  **QAPP** Ensure that all required information on sampling and anal ys is was provided (including planning documents). **Analytes** Ensure that required lists of anal yt es were reported as specified in governing documents (i.e., method, procedure, or contract) **Chain -**  **of -**  **Custody** Examine the traceability of the data from time of sample collection until reporting of data
. Examine chain
- of
- custody records against contract, method, or procedural requirements. **Holding Times** Id entify holding time criteria, and either confirm that they were met or document any deviations
. Ensure that samples were anal yz ed within holding times specified in method, procedure, or contract requirements
. If holding time s were not met, confirm that de viations were documented, that appropriate notifi cations were made (consistent with procedural requirements), and that approval to proceed was received prior to anal ys is. **Sample Handling** Ensure that required sample handling, receipt, and storage procedures were followed, and that deviations were documented. **Sampling Methods and Procedures**  Establish that required sampling methods were used and that deviations were noted
. Ensure that sampling procedures and field measurements met performance criteria and that deviations were documented. **Field Tran scription**  Authenticate transcription accuracy of sampling data (i.e., from field notebook to reports). **Analytical Methods and Procedures**  Establish that required analy tical methods (off

- site laboratory

and on - site anal ytical) were used and that any deviati ons were noted
. Ensure that the QC samples met performance criteria and that any deviations were documented. **Data Qualifiers** Determine that the laboratory data qualifiers were defined and applied as specified in methods, procedures, or contracts. **Laborat ory**  **Transcription** Authenticate accuracy of the transcription of analy tical data (i.e., laboratory

notebook to reporting form, or instrument to L aboratory Information M anagement S ys tems ). **Proficiency Testing** Confirm acceptance of PT sample results against performance requirements as specified in methods, procedures, or contracts. **Standards** Determine that standards are traceable and meet contract, method, or procedural requirements. **Communication** Establish that required communication procedures were followed by field or laboratory

personnel. **Assessment s**  Review field and laboratory assessment reports and accreditation and certification records for performance on specific methods. **Validation Report**
**– Step A**  Summarize deviations from methods, procedure s, or contracts . In clude qualified data and explanation of all data qualifiers.QAPP Handbook D - 8 December 2012 Agency Review Draft **Table D**  **- 4**  **. Validation**  **Step B :**  **Comparison with M QO**  **s Activity**  **Data Deliverables and QAPP**  Ensure that the data report from data validation s tep A was provided. **Deviations** Determine the impacts of any deviations from sampling or anal ytical methods and SOPs
. For example, confirm that the methods given in the QAPP were used and, if they were not, determine if data still meet M QO s . Consider the effectiveness and appropriateness of any corrective action. **Sampling Plan** Determine whether the sampling plan was executed as specified (i.e., the number, location and ty pe of field samples were collected and anal yz ed as specified in the QAPP). **Sampling Procedures** Evaluate whether sampl ing procedures were followed with respect to equipment and proper sampling support (e.g., techniques, equipment, decontamination, volume, temperature, and preservatives) **Collocated Field**  **Duplicates** Compare results of collocated field duplicates with criteria established in the QAPP. **Project Quantification Limits**  Determine that quantitation limits were achieved, as outlined in the QAPP and that the laboratory successfull y analy

zed a standard at the quantitation limit
. **Confirmatory Analyses** Evaluate agreement of laboratory results. **Performance Criteria** Evaluate QC data against project
- specific performance criteria in the QAPP
. **Data Qualifiers** Determine that the data qualifiers applied in S tep A were those specified in the QAPP and that any deviation s from specifications were justified. **Validation Report**
**– Step B**  Summarize the outcome of data comparison to M QOs in the QAPP . In clude qualified data and explanation of all data qualifiers. Further information on some further aspects of verification and validation may be found in Guidance on Environmental Data Verification and Data Validation , QA/G
- 8 (EPA 2008c).QAPP Handbook E - 1 December 2012 Agency Review Draft **APPENDIX E**  **CROSSWALKS TO OTHER DOCUMENTS**  **E.1 CROSSWALK: STANDARDS**  **CLAUSES 7.5 TO 7.10 TO**  **HANDBOOK QAPP**  **AND HANDBOOK**  **QMP** This crosswalk is intended to assist those who are developing a QAPP by linking the Internal and External Standards to this document Handbook QAPP and to the Quality Management Plan Handbook
. **Standards Clause Guidance**  **7.1 Managem**  **ent Representative for Quality** QMP Handbook **7.2 Quality Managem**  **ent Plan** QMP Handbook **7.3 Managem**  **ent Review and Reporting**  QMP Handbook **7.4 Personnel Competence**  QMP Handbook **7.5 Quality Assurance Project Plan**  QAPP Handbook : 1.7 **7.6 Use of Inform**  **ation Technology Methods** QAPP Handbook : 4.2 **7.7 System**  **atic Planning** QAPP Handbook : 2.2, 2.3, 3.2, 3.3, 4.2 **7.8 Assessment of Data And Inform**  **ation** QAPP Handbook : 2.4, 3.4, 4.4 **7.9 Data Review and Usability Reporting**  QAPP Handbook : 2.5, 3.5, 4.5 **7.10 Docum**  **ents and Records Managem ent**  QAPP Handbook : 2.2, 3.2, 4.2 **7.11 Managem**  **ent of Quality Requirements for External Agreem**  **ents** QMP HandbookQAPP Handbook E - 2 December 2012 Agency Review Draft **E. 2**  **CROSSWALK: STANDARDS ANNEX B**  **TO QAPP**  **HANDBOOK** This crosswalk is intended to assist those who are developing a QAPP by linking the S tandard s Annex B to this document QAPP Handbook
.

**Standards Annex B QAPP**  **Handbook Section** 
**B1 Introduction**  1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 and ANSI/ASQ E4
- 2004 1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN? 1.1 0 SUPERSESSION **B2 QAPP Responsibilities and** 

**Application** 1 CHAPTER 1: INTRODUCTION 1.3 THE GRADED APPROACH 1.4 GENERIC QAPPs 1.5 PHASED QAPPs 1. 9 PERIOD OF APPLICABILITY **B2.1 QAPP Preparation** 

**Responsibilities and Approvals** 1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN? 1. 6 WHEN SHOULD A QMP BE COMBINED WITH A QAPP? 1.7 DEVELOPING, REVIEWING, AND APPROVING A QAPP **B2.2 QA Imple**  **mentation and**

**Revision** 1.3 THE GRADED APPROACH 1.4 GENERIC QAPPs 1.5 PHASED QAPPs 1.8 DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP **B2.3 Applicability of QAPPs**  1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 and ANSI/ASQ E4
- 2004 1.9 PERIOD OF APPLICABILITY **B3.1 Overview and General** 

**Requirements** 2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT MEASUREMENT 3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING DATA 4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELSQAPP Handbook E - 3 December 2012 Agency Review Draft **Standards Annex B Handbook**  **Section B3.**  **2 General Content** 

**Requirements** 2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT MEASUREMENT 3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING DATA 4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS **B3. 3**  **Project Managem ent**  2.2 PROJECT MANAGEMENT: DIRECT MEASUREMENT 3.2 PROJECT MANAGEMENT : EXISTING DATA 4.2 PROJECT MANAGEMENT : MODELS **B3. 4**  **Data Acquisition** 2.3 DATA ACQUISITION: DIRECT MEASUREMENT 3.3 DATA ACQUISITION : EXISTING DATA 4.3 DATA ACQUISITION: MODEL DEVELOPMENT, MODIFICATION, AND USE **B3. 5**  **Assessments** 2 .4 ASSESSMENT : DIRECT MEASUREMENT 3.4 ASSESSMENT : EXISTING DATA 4.4 ASSESSMENT : MODEL ASSESSMENT ACTIONS **B3. 6**  **Review ,**  **Evaluation of**

**Usability ,**  **and Reporting Requirements**  2.5 REVIEW, EVALUATION OF USABILITY , AND REPORTING REQUIREMENTS 3.5 REVIEW, EVALUATION OF USABILITY , AND REPORTING REQUIREMENTS 4.5 REVIEW, EVALUATION OF USABILITY: MODEL USABILITY AND REPORTING REQUIREMENTSQAPP Handbook E - 4 December 2012 Agency Review Draft **E. 3**  **CROSSWALK : EPA QA/G** **- 5 TO**  **QAPP HANDBOOK** This crosswalk is provided for those who were familiar with the QA/G
- 5 guidance document. It maps the elements of the obsolete Guidance for Quality Assurance Project Plans, EPA QA/G
- 5 , to this QAPP Handbook. **QA/G -**  **5 QAPP**  **Handbook 1**  **INTRODUCTION** 1.1 AN OVERVIEW OF QUALITY ASSURANCE (QA) PROJECT PLANS 1.6 1.2 EPA POLICY 1.1 1.3 CONTENT OF A QA PROJECT PLAN 1.2 1.4 QA PROJECT PLANS AND THE EPA QUALITY SYSTEM 1.1 1.5 DEVELOPING, REVIEWING, AND APPROVING A QAPP 1.7 1.6 DISTRIBUTING THE QA PROJECT PLAN 1.8 1.7 IMPLEMENTING THE QA PROJECT PLAN 1.8 1.8 RESOURCES Em bedded in each Chapter **2 QA**  **PROJECT PLAN ELEMENTS** Chapter 2 **2. 1**  **Group A: PROJECT MANAGEMENT**  2.2 2. 1 .1 Table of Contents 2.2.1 2. 1 .2 Distribution List 2.2.2 2. 1 .3 Distribution List 2.2.3 2. 1 .4 Project

/Rask Organization 2.2.4 2. 1 .5 Problem Definition/Background 2.2.5 2. 1 .6 Project/Task Description 2.2.5 2.1.7 Quality Objectives and Criteria for Measurement Data 2.2.6 2. 1.8 Special Training Needs/ Certification 2.2.7 2. 1.9 Document s and Records 2.2.8 **2. 2**  **Group B; DATA**  **GENERATION AND ACQUISITION** 2.3 2. 2 .1 Sam

pling Process Design (Experimental Design) 2.3.1 2. 2 .2 Sam

pling Methods 2.3.2 2. 2 .3 Sam

ple Handling and Custody 2.3.3 2. 2 .4 Analytical Methods 2.3.4 2. 2 .5 Quality Control 2.3.5 2. 2 .6 Instrument/Equipment Testing, Inspection, and Maintenance 2.3.6 2.2.7 Instrument/Equipment Calibration and Frequency 2.3.6 2.2.8 Non - direct Measurements Chapter 3 2. 2.10 Data Management 2.3.7 **2. 3**  **Group C: ASSESSMENT**  **AND OVERSIGHT** 2.4 2. 3 . 1 Assessments and Response Actions 2. 4 .1 2. 3 . 2 Reports to Management 2.5.5 2.4. 3 Surveillance of Operations 2.5.4 2.4. 4 Audits of Data Quality 2.4.4 **2. 4**  **Group D: DATA VALIDATION AND USABILITY** 2.5 2. 4 .1 Data Review, Verification and Validation 2.5.1 2. 4 .2 Verification and Validation Methods 2.5.1 2. 4.3 Reconciliation with User Requirements 2.5.4 **3 PROJECTS USING EXISTING DATA**  Chapter 3 3.1 WHEN EXISTING DATA ARE USED ON AN

ENVIRONMENTAL PROJECT 3.1 3. 1 .1 Determine Your Data Needs 3.2.5, 3.3.1 3. 1. 2 Identify Existing Data Sources That Might Meet Project Needs 3.3.1QAPP Handbook E - 5 December 2012 Agency Review Draft **QA/G -**  **5 QAPP**  **Handbook** 3. 1 .3 Evaluate Existing Data Relative to Your Project’s Data Quality

Speci fications 3.4.1, 3.4.2 3. 1 .4 Document Quality Issues in Planning Documents or the Final

Report 3.5.1. 3.5.3 3.2 ISSUES ON PREPARING A QA PROJECT PLAN FOR

PROJECTS USING EXISTING DATA 3.5.2, 3.5.4 **App A BIBLIOGRAPHY**  Standards 11: References Standards Anne x. A: References Standards Anne x. B: References Standards Anne x. C: References **App B GLOSSARY OF QUALITY ASSURANCE AND RELATED** 

**TERMS** Standards 9: Definitions **App C CHECKLIST USEFUL IN QA PROJECT PLAN REVIEW**  Appendix AQAPP Handbook E - 6 December 2012 Agency Review Draft **E. 4**  **CROSSWALK :**  **QAPP HANDBOOK**  **TO EPA QA/G** **- 5**  **, QA/G -**  **5M, AND QA/G -**  **8** This crosswalk is provided for those who were familiar with the QA/G
- 5 guidance document. It maps the elements of this Handbook (CIO
- G - 05 QAPP) to the obsolete Guidance for Quality Assurance Project Plans (QA/G
- 5) , Guid ance for Quality Assurance Project Plans for Modeling (QA/G
- 5M), and Guidance on Environmental Data Verification and Data Validation ( QA/G
- 8)
. **QAPP Handbook**  **Section QA/G** **- 5**  **Section 1**  **INTRODUCTION** Chapter 1 1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM , EPA POLICY

2106 and ANSI/ASQ E4
- 2004 1.2, 1.4 1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN? 1.1, 1.3 1.3 THE GRADED APPROACH 1.2 1.4 GENERIC QAPPs - 1.5 PHASED QAPPs - 1.6 WHEN SHOULD A QMP BE COMBINED WITH A QAPP? 1.1 1.7 DEVELOPING, REVIEWING, AND APPROVING A QAPP 1.5 1.8 DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP 1.6 , 1.7 1. 9 PERIOD OF APPLICABILITY - 1.1 0 SUPERSESSION - **2 QAPP ELEMENTS FOR THE COLLECTION OF DATA BY** 

**DIRECT MEASUREMENT** Chapter 2 2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF

DATA BY DIRECT MEASUREMENT 2 **2.2 PROJECT MANAGEMENT**  (PLAN ) 2.1 Group A 2.2.1 Title, Version, and Approval/Sign
- Off 2.1.1 2.2.2 Document Format and Table of Contents 2.1.2 2.2.3 Distribution List 2.1.3 2.2.4 Project Organization and Schedule 2.1.4 2.2.5 Project Background, Overview , and Intended Use of Data 2.1.5 2.1.6 2.2.6 Data/Project Quality Objectives and Measurement Performance

Criteria 2.1.7 2.2.7 Special Training Requirements and Certification 2.1.8 2.2.8 Documentation an d Records Requirements 2.1.9 **2.3 DATA ACQUISITION**  ( DO) 2.2 Group B 2.3.1 Data Collection Procedure , Experimental Design , and Sampling Tasks 2.2.1 2.3.2 Sam

pling Procedures and Requirements 2.2.2 2.3.3 Sam

ple Handling, Custody Procedures , and Documentation 2.2.3 2.3.4 Analytical Methods Requirements and Task Design 2.2.4 2.3.5 Quality Control Requirements 2.2.5 2.3.6 Instrument/Equipment Testing, Calibration and Maintenance

Requirements , Supplies and Consumables 2.2.6, 2.2.7 2.3.7 Data Management Requirements 2.2.10 **2.4 ASSESSMENTS**  ( CHECK ) 2.3 Group C 2.4. 1 Technical Systems Assessments 2.3.1 2.4. 2 Performance Audits of Measurement and Analytical Systems 2.3.1 2.4. 3 Surveillance of Operations 2.5.4 2.4. 4 Audits of Data Quality 2.4.4 2.4. 5 Qualitative and Quantitative Comparisons to Acceptance Criteria
- 2.4.6 Interim Assessments of Data Quality
- 2.4. 7 Evaluation of Unconventional Measurements
- 2.4. 8 Evaluation of Unconventional Monitoring Projects
-QAPP Handbook E - 7 December 2012 Agency Review Draft **QAPP Handbook**  **Section QA/G** **- 5**  **Section 2.5**  **REVIE W,**  **EVALUATION OF USABILIT Y, AND REPORTING** 

**REQUIREMENTS** (

ACT ) 2.4 Group D 2.5.1 Data Verification and Validation Targets and Methods 2.4.1, 2.4.2 2.5.2 Quantitative and Qualitative Evaluations of Usability
- 2.5.3 Potential Limitations on Data Interpretation
- 2.5.4 Reconciliation with Project Requirements 2.4.3 2.5.5 Reports to Management 2.3.2 **3 QAPP ELEMENTS FOR EVALUATING EXISTING DATA**  Chapter 3 3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING

EXISTING DATA 2.2.9 3.2 **3.2 PROJECT MANAGEMENT**  (PLAN ) 2.1 Group A 3.2.1 Title, Version, and Approval/Sign
- Off 2.1.1 3.2.2 Document Format and Table of Contents 2.1.2 3.2.3 Distribution List 2.1.3 3.2.4 Project Organization and Schedule 2.1.4 3.2.5 Project Background, Overview, and Intended Use of Data 2.1.5, 2.1.6 3.2.6 Data/Project Quality Objectives and Measurement Performance

Criteria 3.1.3 3 .2.7 Special Training Requirements and Certification 2.1.8 3.2.8 Documentation and Records Requirements 3.1.4 **3.3 DATA ACQUISITION**  ( DO) 2.2 Group B 3.3.1 Proposed Data Source Originator and Publication Information 3.1.2 3.3.2 Data Format and Accessibility 2.2.9 3.3.3 Establishment of Acceptance Criteria 3.1.1 3.3.4 Sam

ple Data Collection Methodology
- 3.3.5 Quality Program and Quality Assurance Procedures Used by Data

Originator - 3.3.6 Documentation of Sample Quality Assurance Procedures
- 3.3.6 Data Management Requirements 2.2.10 **3.4 ASSESSMENTS**  ( CHECK ) 2.3 Group C 3.4. 1 Qualitative Comparisons to Acceptance Criteria 3.1.3 3.4. 2 Quantitative Comparisons to Acceptance Criteria 3.1.3 3.4. 3 Assessments to Other Criteria
- 3.4.4 Interim Assessments of Data Quality
- 3.4.5 Evaluation of Unconventional Measurements
- 3.4.6 Evaluation of Unconventional Monitoring Projects
- **3.5**

**REVIEW, EVALUATION OF DATA USABILIT** **Y, AND REPORTING REQUIREMENTS**  **(** ACT ) 2.4 Group D 3.5.1 Data Verification and Validation Targets and Methods 2.4.1, 2.4.2 3.5.2 Quantitative and Qualitative Evaluations of Usability 3.1.4 3.5.3 Potential Limitations on Data Interpretation 3.2 3.5.4 Reconciliation with Project Requirements 2.4.3 3.5.5 Reports to Management 3.1.4 **4 QAPP ELEMENTS FOR**  **DEVELOPMENT, MODIFICATION,**

**AND USE OF MODELS**  EPA QA/G
- 5M 4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS G - 5M Chapter 4 **4.2 PROJECT MANAGEMENT**  (PLAN) G - 5M 4.1 4.2.1 Title, Version, and Approval/Sign
- Off G - 5M 4.1.1 4.2.2 Document Format and Table of Contents G - 5M 4.1.2 4.2.3 Distribution List G - 5M 4.1.3 4.2.4 Project Organization and Schedule G - 5M 4.1.4 4.2.5 Project Background, Overview, and Intended Use of the Model G - 5M 4.1.5QAPP Handbook E - 8 December 2012 Agency Review Draft **QAPP Handbook**  **Section QA/G** **- 5**  **Section** 4.2.6 Data/Project Quality Objectives and Measurement Performance

Criteria G - 5M 4.1.6 4.2.7 Special Training Requirements and Certification G - 5M 4.1.8 4.2.8 Documentation and Records Requirements G - 5M 4.1.9 **4.3 DATA AQCQUISITION: MODEL DEVELOPMENT,** 

**MODIFICATION, AND USE** (DO) G - 5M 4.2 4.3.1 Problem Specification and Identification of Model Purpose and Scope G - 5M 3.3 4.3.2 Model Development or Selection Process G - 5M 4.2.1 4.3.3 Data Requirements for Model Input G - 5M 4.4.2 4.3.4 Evaluation of the Model G - 5M 4.4.1 **4.4 ASSESSMENTS: MODEL ASSESSMENT A**  **CTIONS** (CHECK) G - 5M 4.3 4.4.1 Assessments to Acceptance Criteria and Responses/Corrective Actions G - 5M 4.3.1 4.4.2 Data Management Tasks G - 5M 4.3.2 4.4.3 Model Output Sensitivity and Uncertainty Analysis G - 5M 4.4.1 **4.5 REVIEW** 

**, EVALUATION OF USABILITY: MODEL USABILITY**  **AND REPORTING REQUIREMENTS** (ACT) G - 5M 4.4 4.5.1 Model Evaluation Methods and Activities G - 5M 4.4.1, 4.4.2, 4.4.3 4.5.2 Description of Model Documentation G - 5M 4.1.9 4.5.3 Specifications for Model Maintenance and User Support G - 5M 4.2.3 4.5.4 Reports to Management G - 5M 4.3.2 **App A CHECKLIST OF QAPP ELEMENTS**  Appendix C **App B DATA QUALITY INDICATORS**  - B.1 Sources of Measurement Uncertainty
- B.2 Establishing MQOs in the Context of DQOs
- B.3 Precision
- B.4 Bias
- B.5 Accuracy - B.6 Representativeness
- B.7 Com

parability
- B.8 Com

pleteness
- B.9 Sensitivity
- B.10 The Role of DQIs in the Project Lifecycle
- **App C QUALITY CONTROL SAMPLES**  - **App D DATA REVIEW, VERIFICATION, AND VALIDATION FOR** 

**PROJECT PURPOSES**  EPA QA/G
- 8 D.1 Verification G - 8 2.1, 2.2, 2.3, 2.4 D.2 Validation G - 8 3.1, 3.2, 3.3, 3.4 D.2.1 Validation Activities

– Step A: Compliance G - 8 5.2 D.2.2 Validation Activities

– Step B: Comparison G - 8 5.2 **App E**  **CROSSWALKS** -QAPP Handbook E - 9 December 2012 Agency Review Draft **E.5 CROSSWALK**  **: UFP**  **- QAPP WORKBOOK TO QAP**  **P HANDBOOK**  This crosswalk is intended to assist those who are developing a QAPP using the UFP
- QAPP worksheet format. The sections of this document that provide help for each worksheet of the UFP - QAPP are indicated below.

**UFP -**  **QAPP Worksheet QAPP**  **Handbook Section**  1 & 2 Title and Approval Page 2.2.1 Title, Version, and Approval/Sign
- Off 3 & 5 Project Organization and QAPP Distribution 2.2.3 Distribution List 2.2.4 Project Organization and Schedule 4 , 7 & 8 Personnel Qualifications and Sign
- off Sheet 2.2.1 Title, Version, and Approval/Sign
- Off 2.2.7 Special Training Requirements and

Certification 6 Communication Pathways 2.2.4 Project Organization and Schedule 9 Project Planning Session Summary 2.2.5 Project Background, Overview, and Intended Use of Data 10 Conceptual Site Model 2.2.5 Project Background, Overview, and Intended Use of Data 11 Project

/Data Quality Objectives 2.2.6 Data/Project Quality Objectives and Measurement Performance Criteria 12 Measurement Performance Criteria 2.2.6 Data/Project Quality Objectives and Measurement Performance Criteria 13 Secondary Data Uses and Limitations Chapter 3 QAPP ELEMENTS FOR EVALUATING

EXISTING DATA 14 & 16 Project Tasks & Schedule 2.2.4 Project Organization and Schedule 15 Proje ct Action Limits and Laboratory
S pecific Detection / Quantitation Limits 2.2.6 Data/Project Quality Objectives and Measurement Performance Criteria 17 Sam

pling Design and Rationale 2.3.1 Sam ple Collection Procedure , Experimental Design , and Sampling Tasks 1 8 Sam

pling Locations and Methods 2.3.1 Sam ple Collection Procedure , Experimental Design , and Sampling Tasks 2.3.2 Sam

pling Procedures and Requirements 19 & 30 Sam

ple Containers, Preservation, and Hold Times 2.3.2 Sam pling Procedures and Requirements 20 Field QC Summary 2.3.5 Quality Control Requirements 21 Field SOPs 2.3.2 Sam pling Procedures and Requirements 22 Field Equipment Calibration, Maintenan ce,

Testing, and Inspection 2.3.6 Instrument/Equipment Testing, Calibration and Maintenance Requirements , Supplies and Consumables 23 Analytical SOP s 2.3.4 Analytical Methods Requirements and Task Description 24 Analytical Instrument Calibration 2.3.6 Instrument/Equipment Testing, Calibration and Maintenance Requirements , Supplies and ConsumablesQAPP Handbook E - 10 December 2012 Agency Review Draft **UFP -**  **QAPP Worksheet QAPP**  **Handbook Section**  25 Analytical Instrument and Equipment

Maintenance, Testing, and Inspection 2.3.6 Instrument/Equipment Testing, Calibration and Maintenance Requirements , Supplies and Consumables 26 & 27 Sam

ple Handling, Custody, and Disposal 2.3.3 Sam ple Handling, Custody Procedures , and Documentation 28 Analytical Quality Control and Corrective

Action 2.3.5 Quality Control Requirements 29 Project Documents and Records 2.2.8 Documentation and Records Requirements 3 1 , 3 2 & 33 Assessments and Corrective Action 2.4 ASSESSMENTS AND DATA REVIEW (CHECK) 2.5.5 Reports to Management 34 Data Verification and Validation Inputs 2.5.1 Data Verification and Validation Targets and Methods 35 Data Verification Procedures 2.5.1 Data Verification and Validation Targets and Methods 3 6 Data Validation Procedures 2.5.1 Data Verification and Validation Targets and Methods 37 Data Usability Assessment 2.5.2 Quantitative and Qualitative Evaluations of Usability 2.5.3 Potential Limitations on Data Interpretation 2.5.4 Reconciliation with Project Requirements