Agency Review Draft

Handbook for Developing

Quality Assurance Project Plans

This document is a review draft. This information is distributed solely for the purpose of predissemination review under applicable information quality guidelines. It has not been formally 

disseminated by EPA. It does not represent and should not be construed to represent any Agency

determination or policy. It is being circulated for review of its technical accuracy.
Agency Review Draft

FOREWORD

The U.S. Environmental Protection Agency (EPA) developed the Quality Assurance Project Plan 

(QAPP) as a tool for project managers and planners to document the type and quality of data 

needed to make environmental decisions and to describe the methods for collecting and assessing 

those data. The development, review, approval, and implementation of the QAPP are part of 

EPA’s mandatory Quality Program. The EPA Quality Program requires all organizations to 

develop and operate management structures and processes to ensure that data used in Agency

decisions are of the type and quality needed for their intended use. The QAPP is an integral part 

of the fundamental principles and practices that form the foundation of the EPA Quality

Program.

This Handbook is designed to assist in the creation of QAPPs that address the specifications 

listed in Annex B of Quality Standard For Environmental Data Collection, Production, and Use 

By EPA Organizations (EPA CIO 2106-S-01) and Quality Standard For Environmental Data 

Collection, Production, and Use By Non-EPA (External) Organizations (EPA CIO 2106-S-02.0) 

(current versions). It is intended both for EPA organizations and for organizations conducting 

environmental data operations under external agreements with EPA. 

This document:

 provides technical and policy advice;

 provides implementation information;

 does not substitute for statutes EPA administers or their implementing regulations;

 is not a regulation;

 does not impose legally-binding requirements on EPA, states, or the regulated 

community;

 is not mandatory.

EPA may:

 apply this handbook to any particular program or project to the extent appropriate in light 

of site-specific facts;

 not apply this handbook to a particular situation based upon the specific circumstances;

and

 update, improve, revise, replace or withdraw this handbook at any time.

This document is one of several guidance and handbook documents from the U.S. Environmental 

Protection Agency, Office of Environmental Information (OEI) Quality Staff. EPA Quality

Program documents may be downloaded from the Quality Home Page:

http://www.epa.gov/quality/qa_docs.html

In many instances, regional or program quality assurance personnel and project management 

personnel will be engaged for specific implementation information. These personnel are listed 

here:

http://www.epa.gov/quality/contacts.html

Questions regarding this document or other documents can be directed to the OEI Quality Staff:
Agency Review Draft

U.S. EPA

OEI/Quality Staff (2811R) 

Washington, DC 20460

Phone: (202) 564-6830

FAX: (202) 565-2441

e-mail: quality@epa.govwww.epa.gov/quality
Agency Review Draft

PROLOGUE

The Environmental Protection Agency’s (EPA) mission is to protect human health and to 

safeguard the natural environment — air, water, and land — upon which life depends. EPA 

provides leadership in the nation's environmental science, research, education, and assessment 

efforts. In order to accomplish this mission, EPA must ensure that our efforts meet quality

standards that inspire confidence in our conclusions. The EPA Quality Program provides a 

framework for ensuring that our products and services meet quality standards that are appropriate 

for their intended use.

Quality Assurance Project Plans (QAPPs) are one component of the EPA Quality Program. Each 

project subject to the Internal or External Standards (CIO 2106-S-01.0 and CIO 2106-S-02.0, 

respectively) is required to develop and maintain a QAPP. QAPPs are intended to help a project 

team document how they are going to conduct their project. Each project team has innate ideas 

about the type and quantity of data they will need, how they’ll know if the data are “good 

enough” or not, and how they’ll “double-check” to be sure that their results make sense and are 

meaningful. This document offers some guidance on how to document those ideas. It is meant to 

cover the broad spectrum of activities conducted by the Agency such as setting national 

standards, pollution prevention, permitting, water and air quality, or human health and ecological 

risk assessment. Depending on the organization, the title of the person having quality assurance 

responsibilities can differ from those used in this guidance. For example, a Quality Assurance 

Manager in one organization may be a Quality Assurance Director in another. From the 

contextual use in the guidance, the appropriate person should be recognized.

There are some areas of environmental data that are minimally addressed in this version of the 

Handbook including use of questionnaire surveys and method development studies; later 

versions of the Handbook will address these. 

There are three principal chapters: Chapter 2 is for projects requiring the collection of new data 

to answer the study question; Chapter 3 is for projects making use of existing data; Chapter 4 is 

for projects involving modeling. Each chapter is self contained and should be used in 

conjunction with the Internal and External Standards (specifically, Clauses 7.5 – 7.10, and 

Annex B). Chapters and appendices of greatest utility when creating a QAPP vary with the 

intent and purpose of the data collection:

 New data for a new project: Chapters 1 and 2, Appendices A, B, C, and D

 New data for an established program: Chapters 1 and 2, Appendices A, B, and E

 Using existing data: Chapters 1 and 3, Appendices A, B, and D

 Use in modeling: Chapters 1 and 4, Appendix A

For those familiar with EPA’s previous guidance on QAPPs Guidance for Quality Assurance 

Project Plans, EPA QA/G-5, a crosswalk between that obsolete document and this QAPP 

handbook is provided in Appendix E.

Questions on implementation of EPA’s Quality Management System, including development of 

QAPPs, may be directed to the Quality Staff, Office of Environmental Information

(quality@epa.gov).
v

TABLE OF CONTENTS

CHAPTER 1 INTRODUCTION.................................................................................................1

1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 

and ANSI/ASQ E4-2004..........................................................................................1

1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN? ..................................3

1.3 THE GRADED APPROACH..................................................................................7

1.4 GENERIC QAPPs ...................................................................................................7

1.5 PHASED QAPPs.....................................................................................................9

1.6 WHEN SHOULD A QMP BE COMBINED WITH A QAPP?..............................9

1.7 DEVELOPING, REVIEWING, AND APPROVING A QAPP............................11

1.7.1 Developing a QAPP...................................................................................11

1.7.2 Reviewing a QAPP ....................................................................................12

1.7.3 Approving a QAPP ....................................................................................12

1.8 DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP .................13

1.8.1 Distributing a QAPP ..................................................................................13

1.8.2 Implementing a QAPP ...............................................................................13

1.8.3 Modifying a QAPP ....................................................................................13

1.9 PERIOD OF APPLICABILITY ............................................................................14

1.10 SUPERSESSION...................................................................................................14

CHAPTER 2 QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT 

MEASUREMENT............................................................................................................15

2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA 

BY DIRECT MEASUREMENT...........................................................................15

2.2 PROJECT MANAGEMENT (Plan)......................................................................16

2.2.1 Title, Version, and Approval/Sign-Off......................................................16

2.2.2 Document Format and Table of Contents..................................................17

2.2.3 Distribution List.........................................................................................17

2.2.4 Project Organization and Schedule............................................................18

2.2.5 Project Background, Overview, and Intended Use of Data .......................18

2.2.6 Data/Project Quality Objectives and Measurement Performance 

Criteria .......................................................................................................19

2.2.7 Special Training Requirements and Certification......................................21

2.2.8 Documentation and Records Requirements...............................................21

2.3 DATA ACQUISITION (Do).................................................................................22

2.3.1 Data Collection Procedure, Experimental Design, and Sampling Tasks...23

2.3.2 Sampling Procedures and Requirements ...................................................24

2.3.3 Sample Handling, Custody Procedures, and Documentation ....................25

2.3.4 Analytical Methods Requirements and Task Description .........................28

2.3.5 Quality Control Requirements...................................................................29

2.3.6 Instrument/Equipment Testing, Calibration and Maintenance 

Requirements, QA/QC for Supplies and Consumables.............................30

2.3.7 Data Management Requirements...............................................................32

2.4 ASSESSMENTS (Check)......................................................................................32

2.4.1 Technical Systems Assessments................................................................33

2.4.2 Performance Audits of Measurement and Analytical Systems..................34
vi

2.4.3 Surveillance of Operations.........................................................................35

2.4.4 Audits of Data Quality...............................................................................36

2.4.5 Qualitative and Quantitative Comparisons to Acceptance Criteria ...........37

2.4.6 Interim Assessments of Data Quality ........................................................37

2.4.7 Evaluation of Unconventional Measurements...........................................38

2.4.8 Evaluation of Unconventional Monitoring Projects..................................39

2.5 REVIEW, EVALUATION OF USABILITY, and REPORTING 

REQUIREMENTS (Act)........................................................................................39

2.5.1 Data Verification and Validation Targets and Methods............................40

2.5.2 Quantitative and Qualitative Evaluations of Usability ..............................41

2.5.3 Potential Limitations on Data Interpretation..............................................42

2.5.4 Reconciliation with Project Requirements.................................................42

2.5.5 Reports to Management .............................................................................44

CHAPTER 3 QAPP ELEMENTS FOR EVALUATING EXISTING DATA ......................46

3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING 

DATA ....................................................................................................................46

3.2 PROJECT MANAGEMENT (Plan)......................................................................47

3.2.1 Title, Version, and Approval/Sign-Off......................................................47

3.2.2 Document Format and Table of Contents..................................................48

3.2.3 Distribution List.........................................................................................48

3.2.4 Project Organization and Schedule............................................................49

3.2.5 Project Background, Overview, and Intended Use of Data .......................49

3.2.6 Data/Project Quality Objectives and Measurement Performance 

Criteria .......................................................................................................50

3.2.7 Special Training Requirements and Certification......................................51

3.2.8 Documentation and Records Requirements...............................................51

3.3 DATA ACQUISITION (Do).................................................................................52

3.3.1 Proposed Data Source Originator and Publication Information ................52

3.3.2 Data Format and Accessibility...................................................................52

3.3.3 Establishment of Acceptance Criteria........................................................53

3.3.4 Sample Data Collection Methodology.......................................................54

3.3.5 Quality Program and Quality Assurance Procedures Used by Data 

Originator...................................................................................................54

3.3.6 Documentation of Sample Quality Assurance Procedures........................55

3.3.7 Data Management Requirements...............................................................55

3.4 ASSESSMENTS (Check)......................................................................................56

3.4.1 Qualitative Comparisons to Acceptance Criteria.......................................56

3.4.2 Quantitative Comparisons to Acceptance Criteria.....................................62

3.4.3 Assessments to Other Criteria....................................................................63

3.4.4 Interim Assessments of Data Quality ........................................................63

3.4.5 Evaluation of Unconventional Measurements...........................................63

3.4.6 Evaluation of Unconventional Monitoring Projects..................................64

3.5 REVIEW, EVALUATION OF USABILITY, AND REPORTING 

REQUIREMENTS (Act) .......................................................................................64

3.5.1 Data Verification and Validation Targets and Methods............................64

3.5.2 Quantitative and Qualitative Evaluations of Usability ..............................64
vii

3.5.3 Potential Limitations on Data Interpretation..............................................65

3.5.4 Reconciliation with Project Requirements.................................................65

3.5.5 Reports to Management .............................................................................67

CHAPTER 4 QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, AND 

USE OF MODELS...........................................................................................................69

4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS.........................................69

4.2 PROJECT MANAGEMENT (Plan)......................................................................70

4.2.1 Title, Version, and Approval/Sign-Off......................................................72

4.2.2 Document Format and Table of Contents..................................................72

4.2.3 Distribution List.........................................................................................73

4.2.4 Project Organization and Schedule............................................................73

4.2.5 Project Background, Overview, and Intended Use of the Model ..............74

4.2.6 Data/Project Quality Objectives and Acceptance Criteria.........................78

4.2.7 Special Training Requirements and Certification......................................79

4.2.8 Documentation and Records Requirements...............................................80

4.3 DATA ACQUISITION: MODEL DEVELOPMENT, MODIFICATION, 

AND USE (Do)......................................................................................................80

4.3.1 Data Requirements for Model Input ..........................................................81

4.3.2 Data Management Tasks............................................................................83

4.4 ASSESSMENTS: MODEL EVALUATION ACTIONS (Check) ........................84

4.4.1 Model Evaluation Methods and Activities ................................................84

4.4.2 Evaluation of the Model................................................................................85

4.4.3 Model Output Sensitivity and Uncertainty Analysis.................................87

4.5 REVIEW, EVALUATION OF USABILITY: MODELING RESULTS 

USABILITY AND REPORTING REQUIREMENTS (Act) ................................87

4.5.1 Peer Review ...............................................................................................88

4.5.2 Description of Model Documentation .......................................................89

4.5.3 Specifications for Model Maintenance and User Support .........................90

4.5.4 Reports to Management .............................................................................90

GLOSSARY..................................................................................................................................91

REFERENCES.............................................................................................................................97

APPENDIX A CHECKLIST OF QAPP ELEMENTS ...........................................................A1

APPENDIX B DATA QUALITY INDICATORS...................................................................B1

B.1 Sources of Measurement Uncertainty...................................................................B3

B.2 Establishing MQOs in the Context of DQOs........................................................B3

B.3 Precision................................................................................................................B5

B.4 Bias .......................................................................................................................B5

B.5 Accuracy ...............................................................................................................B6

B.6 Representativeness................................................................................................B7

B.7 Comparability .......................................................................................................B9

B.8 Completeness......................................................................................................B10

B.9 Sensitivity ...........................................................................................................B11

B.10 The Role of DQIs in the Project Lifecycle .........................................................B14
viii

APPENDIX C QUALITY CONTROL SAMPLES ................................................................C1

APPENDIX D DATA REVIEW, VERIFICATION, AND VALIDATION FOR 

PROJECT PURPOSES..................................................................................................D1

D.1 Verification ...........................................................................................................D4

D.2 Validation..............................................................................................................D5

D.2.1 Validation Activities – Step A: Compliance.............................................D6

D.2.2 Validation Activities – Step B: Comparison.............................................D6

APPENDIX E CROSSWALKS TO OTHER DOCUMENTS...............................................E1

E.1 Crosswalk: Standards CLauses 7.5 to 7.10 to Handbook QAPP and 

HANDBOOK QMP..............................................................................................E1

E.2 Crosswalk: Standards ANNEX B to QAPP Handbook ........................................E2

E.3 Crosswalk: EPA QA/G-5 to QAPP Handbook.....................................................E4

E.4 Crosswalk: QAPP Handbook to EPA QA/G-5, QA/G-5M, and QA/G-8 ............E6

E.5 Crosswalk: UFP-QAPP Workbook to QAPP HANDBOOK ...............................E9
ix

ACRONYM LIST

ANSI American National Standards Institute

ASTM American Society for Testing and Materials

ASQ American Society for Quality

CFR Code of Federal Regulations

CIO Chief Information Officer

DL Detection Limit

DQA Data Quality Assessment

DQI Data Quality Indicator

DQO Data Quality Objective

EPA Environmental Protection Agency

FIFRA Federal Insecticide, Fungicide, and Rodenticide Act

FORMS Field Operations and Records Management System

GPS Global Positioning System

ICP Inductively Coupled Plasma

IQG Information Quality Guidelines

ISO International Organization for Standardization (not an acronym)

IT Information Technology

LOD Limit of Detection

LOQ Limit of Quantitation

MCL Maximum Contaminant Level

MDL Method Detection Limit

MPC Measurement Performance Criteria

MQO Measurement Quality Objective

OEI Office of Environmental Information

OMB Office of Management and Budget

ORD Office of Research and Development

OSWER Office of Solid Waste and Emergency Response

PCB Polychlorinated biphenyls

PE Performance Evaluation

PQL Practical Quantitation Limit

PQO Project Quality Objectives

PRP Potentially Responsible Parties

PT Proficiency Test
x

QA Quality Assurance

QAPP Quality Assurance Project Plan

QC Quality Control

QL Quantitation Limit

QMP Quality Management Plan

QMS Quality Management System

RCRA Resource Conservation and Recovery Act 

SOP Standard Operating Procedure

TSA Technical Systems Assessment

TSCA Toxic Substances Control Act

UFP Uniform Federal Policy

XRF X-ray Fluorescence
QAPP Handbook 1 December 2012 Agency Review Draft

CHAPTER 1

INTRODUCTION

1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 AND 

ANSI/ASQ E4-2004

This Chapter lays a general foundation for a QAPP and provides a more detailed explanation of 

what is required in the Internal and External Standards. The Chapter discusses key aspects of 

what a QAPP contains, what it applies to, its connection to a QMP, and who is responsible for 

developing one. It lays out the general alignment of required content in a Plan-Do-Check-Act 

paradigm. 

This document presents non-mandatory guidance intended to help its users prepare Quality 

Assurance Project Plans (QAPPs) that meet the requirements of the following two 

Environmental Protection Agency (EPA) Standards:

 Quality Standard for Environmental Data Collection, Production, and Use by EPA 

Organizations, (CIO 2106-S-01), “Internal Standard” (EPA 2013a); and

 Quality Standard for Environmental Data Collection, Production, and Use by Non-EPA 

(External) Organizations, (CIO 2106-S-02), “External Standard” (EPA 2013b).

These Standards provide the foundation for the Agency-wide Quality Program for environmental 

data-related products and services that are disseminated outside the Agency. They conform to 

EPA Quality Policy, CIO 2106.0, “Quality Policy” (EPA 2008a) and to EPA Procedure for 

Quality Policy, CIO 2106-P-01.0, “Quality Procedure” (EPA 2008b) as issued by the Chief 

Information Officer (CIO), EPA. The use of this Handbook is discretionary but should be used in 

conjunction with the two EPA Quality Standards. Throughout Chapters 2, 3, and 4 of this 

document, the corresponding sections of the Internal and External Standards are noted in italics. 

This handbook follows the lead of the Standards in reaffirming the applicability of the national 

consensus standard, Quality Systems for Environmental Data and Technology Programs: 

Requirements with Guidance for Use, ANSI/ASQ E4-2004 (ANSI 2004) developed by the 

American National Standards Institute (ANSI) and the American Society for Quality (ASQ).

The QAPP document Requirements for Quality Assurance Project Plans, EPA QA/R-5 (EPA 

2001a), has been withdrawn in favor of these Internal and External Standards. The QA/R-5 

document should not be used for developing QAPPs except where explicitly allowed by the 

Standards. The guidance that accompanied QA/R-5, Guidance for Quality Assurance Project 

Plans, EPA QA/G-5, has also been withdrawn as it is superseded by this document, EPA 

Guidance on Quality Assurance Project Plans (CIO 2106-G-05 QAPP), “QAPP Guidance”. It is 

important to remember that unlike the Standards, which present normative requirements, this 

document consists of informative (non-mandatory) guidance presented in non-prescriptive 

language. It is offered as a helpful resource for those who need to prepare QAPPs that comply

with EPA Standards. It may be helpful to those creating work plans, sampling and analysis plans, 

field implementation plans, or other project planning documents. 
QAPP Handbook 2 December 2012 Agency Review Draft

EPA also allows use of the Uniform Federal Policy for Quality Assurance Project Plans, (EPA505-B-04-900A), “UFP-QAPP” (EPA 2005a) as an alternative approach to satisfy EPA’s 

requirement for a QAPP if allowed by the external agreement. In 2005, the Intergovernmental 

Data Quality Task Force (IDQTF) issued the UFP-QAPP to address environmental data 

collection, production, and use for hazardous waste cleanup activities at federal facilities and 

installations1. Shortly thereafter, EPA’s Office of Solid Waste and Emergency Response issued a 

directive (OSWER Directive 9272.0-17) (EPA 2005c) stating that the UFP-QAPP format should 

be used for EPA-managed Federal Facility projects where environmental data are collected and 

used. Both that directive and further guidance issued jointly by EPA’s Federal Facilities and 

Restoration Reuse Office and Quality Staff, Office of Environmental Information (OSWER 

9272.0-20) (EPA 2005d) state that QAPPs prepared and approved according to the UFP will be 

considered consistent with EPA Standards. The UFP-QAPP worksheets are also consistent with 

the EPA Quality Standards. 

Figure 1 depicts the hierarchical relationship of QA documents: ANSI/ASQ E4-2004 national 

consensus standard, together with the Information Quality Act of 2001 (IQG 2001), establish a 

basis for the Agency’s Quality Policy; the Quality Procedure provides additional explanation 

about how to carry out the Policy; the two Standards contain requirements for applying the 

Policy and Procedure to environmental data operations (within and external to the Agency);

Handbook for Preparing Quality Management Plans (EPA 2013c) offers help in preparing

Quality Management Plans (QMPs) that meet the requirements of the higher level documents; 

and this Handbook document offers guidelines, advice, and examples that will help users satisfy

provisions of the Standards at the project level. 

Figure 1. Relationship among EPA Quality Program Documents

 

1

The IDQTF consists of representatives from EPA, the Department of Defense (DoD), and the Department of Energy (DOE). It 

was established to address real and perceived inconsistencies and deficiencies in quality control for laboratory data, within and 

across governmental organizations, which result in greater costs, time delays, and increased risk. It is chaired by the Director of 

EPA’s Federal Facilities Restoration and Reuse Office and operates as a partnership, reaching decisions through consensus.

Handbook for Preparing Quality

Management Plans 

Quality Procedure 

CIO Procedure 2106-P-01.0

Handbook for Developing Quality 

Assurance Project Plans 

Internal Standard

CIO 2106-S-01.0

External Standard 

CIO 2106-S-02.0

Quality Policy CIO 2106

National Consensus 

Standard ANSI/ASQ E4-

2004

Information Quality Act
QAPP Handbook 3 December 2012 Agency Review Draft

1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN?

This section describes QAPPs in a Question-and-Answer format and explains more fully the 

requirements of Standard 7.5, Quality Assurance Project Plan, and Annex B1, Introduction, B2,

QAPP Responsibilities and Approvals, B2.3, Applicability, and B3.2, General Content 

Requirements.

A QAPP is a formal document describing in comprehensive detail the necessary quality

assurance (QA), quality control (QC), and other technical activities that will be implemented to 

ensure that the results of the work performed will satisfy the stated performance criteria. It

describes the activities of a project in the acquisition of environmental data or information from 

direct measurement activities, existing data, or generated by models.

What is the difference between QA and QC? QA refers to the system of management activities

that are designed to ensure quality, whereas QC refers to the system of preventive and corrective 

technical activities that measure or monitor performance against defined standards and makes 

adjustments to maintain acceptable performance. 

Why is a QAPP important? A QAPP presents the steps that should be taken to ensure that 

environmental data collected, produced, and used are of adequate and sufficient type and quality 

required for a specific decision or use. It presents an organized and systematic description of the 

ways in which QA and QC will be applied to the collection and use of environmental data. It 

documents the results of the project’s planning process thus providing a concise record of the 

conduct of environmental data operations.

Must I have a QAPP? ANSI/ASQ E4-2004 Section 6 (Part B) (ANSI 2004) and the EPA Internal 

and External Standards, Clause 7 (EPA 2013a, EPA 2013b) require that a QAPP be approved for 

all data collection projects. The specific requirements for the content of a QAPP are presented in 

Annex B of each Standard.

What are the benefits of a QAPP? A QAPP (or equivalent document as stipulated in the 

organization’s QMP) integrates QA and QC aspects of a project throughout its life cycle, 

including planning, implementation, assessment, and corrective actions. A QAPP is a “roadmap” 

for ensuring the implementation of quality procedures and practices in a project. When 

implemented as prescribed, a QAPP increases efficiency and enables decisions to be made 

effectively with greater certainty and reduces the potential for possible rework. 

To what does a QAPP apply? The ultimate success of an environmental program or project 

depends on the quality of the environmental data collected and used in decision-making, and this 

may depend significantly on the adequacy of the QAPP and its effective implementation.2The 

title of the QAPP being developed should include information as to what the QAPP will apply to 

and bear a unique identifying number (or QAPP Tracking Number) such that it can be readily 

identified amongst similar QAPPs.

 

2

This paragraph is taken verbatim from the introduction to the Workbook for Uniform Federal Policy for Quality Assurance 

Project Plans (EPA-505-B-04-900C) (EPA 2005b). Throughout the QAPP Guidance there are several instances of shared 

language with the UFP-QAPP. This serves as a general notice that these documents are consistent by design, and specific 

citations will not be provided.
QAPP Handbook 4 December 2012 Agency Review Draft

Agency activities involving environmental data required by the Internal and External Standards

to have a QAPP include, but are not limited to:

 direct and indirect field and/or laboratory measurements;

 evaluating the operation and performance of environmental technology;

 inspections;

 existing environmental data;

 questionnaire survey development or application;

 development and validation of sampling or analytical methods;

 environmental model modification and/or development;

 enforcement monitoring or assessments;

 application of environmental management systems;

 environmental safety and health monitoring;

 scientific research;

 regulatory development;

 statistical or economic analyses of environmental data;

 use of information technology (mathematical models);

 use of information sources outside of direct EPA management controls or authority; and

 use of other data sources (e.g., literature or the Internet). 

In some instances an organization may develop a QAPP for field collection activities and another 

for laboratory analysis activities; a clear distinction between them is advisable. For projects with 

multiple participants each developing a QAPP, an overarching QAPP which would provide

requirements such that individual components may be effectively integrated is advised.

Who is involved in developing a QAPP? Most activities involving environmental data require the 

coordinated efforts of many individuals, possibly including managers, engineers, scientists, 

statisticians, information technology (IT) experts, modelers, and others. The QAPP should

integrate the contributions and requirements of everyone involved into a clear, concise statement 

of what needs to be accomplished, how it will be done, and by whom. It should provide 

understandable instructions to those who must approve or implement the QAPP, including the 

field sampling team, the analytical laboratory, and data users and reviewers. Beyond the general 

guidance provided in the QAPP, it should identify and integrate the use of related policies and 

procedures applicable to the project such as administrative procedures, laboratory procedures, 

data analysis methodologies, IT policies and procedures, and data handling and analysis policies.

How is the QAPP effective? The QAPP must specify the level or degree of QA and QC activities 

needed for the particular environmental data or model operations with clear objectives, 

acceptance criteria, and QA/QC control strategies. The QA and QC technical requirements of a 

project should be commensurate with the type of work to be done (e.g., monitoring, site 
QAPP Handbook 5 December 2012 Agency Review Draft

characterization, model simulation, and bench level proof of concept), the purpose of the project 

(e.g., regulatory enforcement, development work for rulemaking, permit approval, research 

publications, and journal articles), and the scale of the project such as a one-time assessment or a 

template for multiple assessments. Regardless of the complexity of the project, the QAPP 

documents how the project team will ensure that the quality of data is suitable for its intended 

use by documenting acceptance criteria against which assessments may be made .

Why is it important to me? The QAPP is invaluable in not only documenting all aspects of the 

project serving as a structured resource for writing the final report on the project. In addition, it 

helps a successful implementation during reassignments of personnel. It is the basis for assuring 

the integrity of the project’s conclusions.

Who approves a QAPP before being put into operation? Each QAPP must be approved by the 

organization’s project manager or director and the QA Manager prior to any data collection, 

production, or use (with the possible exception of circumstances requiring immediate action to 

protect human health and the environment, or operations conducted under police powers). If this 

approval function has been delegated, it must be in accordance with the organization’s QMP.

How do I develop a QAPP? A brief summary of the process:

1. determine the scope and objectives of the project and gather background information;

2. assemble a project team with necessary project and QA expertise;

3. conduct planning sessions to determine how to collect the specific type of data, the 

amount of data, and what the goals or acceptance criteria will be;

4. using the Standard and Handbook (if necessary) develop a draft QAPP;

5. circulate the draft QAPP for peer review, comment, and input for improvement;

6. submit the final QAPP for approval;

7. after approval, distribute the QAPP to all persons involved with the project; and

8. proceed to implement the QAPP allowing for documentation of changes, re-approvals, 

and distribution of updated QAPPs.

How is a QAPP organized? The QAPP elements are arranged in four sections to reflect the 

logical flow of the project life cycle (also known as the Plan-Do-Check-Act steps, Figure 2).

Figure 2. Plan-Do-Check-Act Project Life Cycle
QAPP Handbook 6 December 2012 Agency Review Draft

The four types of elements in these steps are: 

 Project Management (Plan) - These elements, covering the basic areas of project 

management, project objectives, and the roles and responsibilities of the participants, 

document the development of progress towards the project’s goal. The development of 

technical and quality project objectives, such as Data Quality Objectives (DQOs), are part 

of this step.

 Data Acquisition (Do) - These elements cover all aspects of data measurement, data 

acquisition designs, and their implementation. The development of performance and 

acceptance criteria for the collection of data by direct measurement, acquisition from 

existing sources, and by modeling are integral to this step. This ensures the intended 

measurements, data collection, or acquisition methods are appropriate for achieving 

project objectives.

 Assessments (Check) - These elements address the activities for assessing the 

effectiveness of the implementation of the project and associated QA and QC activities. 

The purpose of assessment is to ensure that the QAPP is implemented as prescribed and 

that project actions are implemented as expected.

 Review, Evaluation of Usability, and Reporting Requirements (Act) - These elements 

cover the QA activities that occur after the data collection (or use) phase of the project is 

completed. Implementation of these elements will help in determining that the data 

conform to the specified criteria, and ensuring that data usability can be documented in a 

final project report. This step includes any limitations or restrictions on the use of the 

data, or contingencies for revising project objectives.

How is this QAPP Handbook organized? This document covers QAPP development for three 

different types of projects:

 studies in which new data will be collected (Chapter 2); 

 investigations in which existing data will be used (Chapter 3); and 

 development, modification, and use of models using environmental data (Chapter 4).

These three project types are each addressed in their own chapter and should be used with the 

relevant appendices of this document. It is not the intent of this Handbook to specifically identify 

every type of project for which a QAPP will be needed, but rather to describe QAPP elements for 

general types of projects. If a QAPP element is not applicable, then an indication of why it is not 

relevant should be given. 

If information is available elsewhere should I rewrite this into the QAPP? Reference to existing 

documents (such as standard operating procedures, sampling and analysis plans, environmental 

assessments, work plans, literature files, fully documented models) can greatly reduce the size of 

a QAPP. Documents may be included by reference or appended to the QAPP. It is essential, 

however, that everyone, including QAPP reviewers, involved with the project has access to the 

referenced documents.
QAPP Handbook 7 December 2012 Agency Review Draft

How long is a QAPP? The QAPP needs to have enough information to adequately describe the 

project’s objectives, implantation, and results. This implies a complex project will demand more 

than a relatively simple study. The use of a “graded approach” is recommended.

Is guidance written by other organization based on EPA QA/G-5 obsolete? All of the strengths 

of the now obsolete guidance EPA QA/G-5 (EPA 2002a) have been incorporated into this QAPP 

guidance. This means that guidance written by other organizations based on the obsolete 

document will still be very strong, but will not have some of the latest advances in QA practices.

Organizations are encouraged to review their guidance to ensure it meets the requirements of the 

Standards, and also take advantage of some of the advice offered in this QAPP guidance. 

1.3 THE GRADED APPROACH

The graded approach is an extremely important concept and the outline given in Standards 

Annex B3.1, General Requirements, amplified by discussion.

Recognizing that a “one size fits all” approach to quality requirements will not work in 

organizations as diverse as EPA, implementation of the EPA Quality Program is based on the 

principle of a graded approach. The graded approach is the process of establishing the project 

requirements and level of effort commensurate with the intended use of the results and the 

degree of confidence needed in the quality of the results.

The graded approach allows for the content and level of detail in each QAPP to vary

accordingly. For example, the quality expectations of fundamental research may be different

from that of regulatory compliance because the purpose or intended use of the data is different. 

The intent of the graded approach is to achieve sufficient detail in the QAPP to satisfy the 

objectives of the project, ensure adequate quality for the intended use of the data, and meet the 

resources allocated to the project. Cooperation and discussion during the QAPP development 

process is encouraged, especially on the implementation of the graded approach. The final 

decision on QAPP content and level of detail belongs to the EPA organization responsible for the 

work to be done, consistent with the approved QMP, and within the resources available for the 

project.

1.4 GENERIC QAPPs

Many organizations are engaged in projects or programs collecting data under very similar 

circumstances. This section offers advice on the use of generic QAPPs using the requirements of 

Standards Clause 7.5, QAPP, and Annex B3, General Requirements.

A generic QAPP provides an overarching plan that describes the quality objectives and 

documents a comprehensive set of sampling, analysis, QA/QC, data review, and assessment 

procedures specific to a large program or long-term project. In contrast to the project specific 

QAPP, the generic QAPP serves as an umbrella under which multiple data collection, production 

and use activities may be conducted over an extended period of time. 

Generic QAPPs may make sense in situations such as where multiple sites, systems, or projects

will be sampled under a common sampling and analysis protocol/plan. A simple way to 

determine whether a generic QAPP is appropriate for a project is to consider whether there is 
QAPP Handbook 8 December 2012 Agency Review Draft

sufficient consistency across the QA needs of multiple projects within a program that combining 

the planning into a single, generic QAPP will:

 ensure the necessary level of quality for all projects covered by the generic QAPP; and

 require less time and resources to manage with a single generic QAPP than with multiple 

project specific QAPPs.

An approved generic QAPP should be supported by task or project specific addenda, which 

address the issues unique to each task or project. Project or task specific information that is not 

covered by the generic QAPP should be documented in detail in these addenda. The generic 

QAPP should specify the preparation, review, and approval of task or project specific addenda. 

For external non-EPA projects, EPA may authorize the organization to approve project specific 

addenda contingent upon a review and approval process that is fully documented in the approved 

generic QAPP.

Some organizations may find a variant on generic QAPPs, developed by EPA Region 9, more 

suitable to their needs. Called Region 9 QA Program Plans,

“...they are intended to establish policies that define and document the type and 

quantity of data needed for program level environmental decisions and to describe 

the methods required for collecting, analyzing, and assessing data to support those 

decisions. For the purposes of this [EPA Region 9] guidance, an environmental 

program is considered to be a series of activities which are based directly or 

indirectly on an act of Congress and defined in regulations promulgated by EPA, 

State, or tribal governments...The measurements under a program generally 

reflect on-going activities which do not have defined start and end dates (not to be 

confused with grant cycles). This does not preclude a program implementing or 

conducting projects. Program activities are usually of a recurring nature although 

specific activities may not recur. For example, there may be on-going water 

monitoring, but sampling locations may change from year to year. EPA and/or a 

funded organization may sponsor environmental projects. Projects are considered 

to be activities that have a definable beginning and a definable end. Projects 

would be expected to have their own QAPPs that would either be approved by

EPA or by the other organization. A QA Program Plan should describe policies 

regarding what type of grants, cooperative agreements, or activities require a 

QAPP or other QA documentation, what the nature of that documentation would 

be, and what activities might be exempted from that requirement. For example, 

an investigation of an integrated pest management approach under FIFRA or an 

evaluation of a new wastewater treatment methodology would be considered 

projects that included environmental measurements that would be under a broader 

environmental program. A QA Program Plan should also describe what 

information the QAPP should include or cite appropriate guidance...Both QA 

Program Plans and QAPPs should include specific identifiable goals and 

objectives and discuss the uses of the data. Often, but not always, program goals 

are more closely tied to environmental regulations, whereas project goals are 

developed for that event based on a systematic planning ” (EPA 2012). 
QAPP Handbook 9 December 2012 Agency Review Draft

1.5 PHASED QAPPs

Some organizations need QAPPs that are brought on-line as a project is being developed. This

section offers advice on the use of a set of QAPPs using Standards 7.5, QAPP.

When there is significant uncertainty surrounding a project and additional clarity is expected as 

initial data or information are gathered, or for a project that is iterative in nature, it may be 

appropriate to use a phased approach to QAPP development. The phased QAPP should include a 

description of the decision points for the project. Examples of when a phased QAPP might be 

appropriate include:

 literature reviews in which the eventual direction and depth of the research is dependent 

upon the information found in the articles in the first round of the project;

 investigations or phased project cycles where the results of the initial assessment will 

dictate the following steps (e.g., no contaminants found above thresholds leading to site 

closure vs. contaminants found above thresholds leading to a risk assessment and 

feasibility study);

 analytical method development for a specific purpose (e.g., improved sensitivity) for 

which an early step in the process may determine the viability of continued effort; or

 model development and implementation for which the project overview, objectives, and 

measurement performance criteria can be defined upfront, but until the model 

requirements are defined, the requirements for input data can’t be clearly established.

If a phased QAPP is developed, it should still contain all relevant elements of a QAPP. The 

efficiency to be gained from a phased QAPP may be realized by not spending time or resources 

to specify the detailed project requirements for each “what-if” scenario for the project. If a 

phased QAPP is developed, it is very important that it be updated as the direction of the project 

becomes clear. No project work should take place without an appropriate QAPP in place, so 

moving beyond the phase documented in the QAPP should not occur until the QAPP is updated. 

These updates should be described in the revision history (see Section 1.8). 

1.6 WHEN SHOULD A QMP BE COMBINED WITH A QAPP?

Some organizations have no direct need for a full Quality Management Plan (QMP) but could 

incorporate needed elements into a combined QMP/QAPP outlined in Standards Clause 7.5.2. 

The role of a QMP is to define and describe an organization’s overall structure, processes, roles,

and responsibilities for planning, implementing, and evaluating environmental data operations. 

The role of a QAPP is to provide a road map for environmental data production, collection, and 

use in a project. QAPPs document project objectives along with the QA and QC procedures that 

will ensure that those objectives are achieved. While the QMP defines the Quality Program at the 

organizational level, the QAPP defines project-specific quality concerns. There may be, and 

usually are, multiple QAPPs developed under a QMP. Figure 3 depicts the relationship between 

QMPs and QAPPs within an organization’s Quality Management System (QMS). QAPPs are 

needed for projects where collection of new data is planned, existing data will be incorporated to 

meet project objectives, or models will be used to evaluate and interpret data. 
QAPP Handbook 10 December 2012 Agency Review Draft

Figure 3. Relationship between QMP and QAPPs

In some circumstances, EPA may recommend the preparation of a combined QMP and QAPP as 

an appropriate means to document organizational level and project-level environmental data 

activities. The goal of the project, together with its data needs, is the principal driver of the 

QA/QC activities documented in the combined QMP/AQPP. Examples where this may be 

appropriate could include research projects, grants, inspection programs, and monitoring 

networks involving multiple small organizations. In such cases, a combined QMP and QAPP can 

be an effective means to document organizational level and project-level environmental data 

activities, particularly when a full QMP is not needed. Combined QMP/QAPPs should not be 

used for external agreements containing multiple projects that may be determined over the life of 

the agreement such as mission contracts and program grants). Some specific situations in which 

a combined QMP/QAPP might be appropriate include (but are not limited to):

 routine inspection programs where inspections are conducted in the same manner for all 

visits (e.g., EPA lead or radon programs);

 small organizations that do infrequent work with EPA funds (e.g., competitively awarded 

one-time grants and environmental education grants);

 one-time mobilization to a previously unvisited field location such as for monitoring 

assistance to a Tribal location;

 necessary quick-turnaround data analyses that may involve new laboratory facilities, 

methods, or equipment;

 environmental measurements related to technology performance, where QMS planning 

for the operation of the technology is already addressed by another document;

 academic research activities of a one-time nature where an organizational QMP has more 

detail than needed; or

 a monitoring program conducted by a volunteer organization where the size and structure 

of the organization is limited.

A combined QMP/QAPP incorporates some elements of the QMP and some elements of the 

QAPP. A combined QMP/QAPP is primarily a QAPP; that is, it pertains to a specific project or 
QAPP Handbook 11 December 2012 Agency Review Draft

program. However, the project may not require the application of a full QMS. As with any QMP, 

it should document the management controls that should be applied to the planning, 

implementation, and assessment of quality activities, and what processes will be followed to 

carry out those activities. These may vary depending upon the scope and objectives of the project 

and should be defined by the EPA organization.

It is EPA’s responsibility to ensure that all elements of a combined QMP/QAPP, as defined by 

the EPA organization, are addressed by the organization performing the work. The combined 

QMP/QAPP is subject to the same EPA review and approval as any other QAPP. The QMP 

elements of the combined QMP/QAPP should be subject to appropriate QMP-level review.

Further guidance for a combined QMP/QAPP is provided in the QMP Guidance.

1.7 DEVELOPING, REVIEWING, AND APPROVING A QAPP

This section offers guidance on Standards Clause 7.5 QAPP, and Annex B2.1, QAPP 

Preparation Responsibilities and Approvals, and Annex B2.2 QAPP Implementation and 

Revision.

1.7.1 Developing a QAPP

The QAPP may be prepared by an EPA organization or a non-EPA organization (e.g., a 

contractor, an assistance agreement holder, another Federal agency under an interagency

agreement). All EPA organizations collecting, producing, or using environmental data must have 

their QAPPs approved by their managers and QA Manager or designee as specified in the 

organization’s QMP. Except where specifically delegated in the QMP of the EPA organization 

sponsoring the work, all QAPPs prepared by non-EPA organizations must be approved by EPA 

before implementation, and involvement of EPA and other regulatory agencies, as applicable, in 

development is encouraged.

QAPP development necessitates the coordinated efforts of many individuals such as those who 

will gather data and those who will use the data to make decisions. These individuals include: 

decision makers, project managers, regulators, stakeholders, and technical staff (e.g., 

hydrologists, biologists, chemists, samplers, statisticians, engineers, economists, modelers, and 

risk assessors). In addition, peer reviewers and individuals with varied expertise ensure that 

technical areas are sufficiently addressed, thus helping to identify important issues during 

review, approval, and implementation. 

For internal EPA projects, the Project Manager (PM) or Principal Investigator (PI) is generally

responsible for overseeing the preparation of the QAPP. For external projects, the funded entity

is usually responsible for QAPP development (grantees being allowed to develop an independent 

QAPP), but for other agreements (for example, contractual) the QAPP is subject to review and 

approval by EPA prior to implementation. In the case of another Federal agency receiving funds 

from EPA, as through an interagency agreement, the award indicates who is responsible for 

QAPP preparation. When EPA receives project funds from another Federal agency, EPA 

personnel usually write the QAPP unless the other organization negotiates specific quality 

requirements. It is usually advantageous to hold round-table discussions between stake-holders, 

QA specialists, and Project Manager prior to writing the QAPP. This ensures the coverage of the 

QAPP is inclusive and has a broad coverage of the project.
QAPP Handbook 12 December 2012 Agency Review Draft

Chapters and appendices of greatest utility when creating a QAPP vary with the intent and 

purpose of the data collection:

 New data for a new project: Chapters 1 and 2, Appendices A, B, C, and D

 New data for an established program: Chapters 1 and 2, Appendices A, B, and E

 Using existing data: Chapters 1 and 3, Appendices A, B, and D

 Use in modeling: Chapters 1 and 4, Appendix A

1.7.2 Reviewing a QAPP

A QAPP can be reviewed by several authorized EPA reviewers to ensure it contains the 

appropriate content and level of detail as defined by the EPA organization’s QMP. These 

reviewers may include the Project Manager, Contracting Officer’s Technical Representative, and 

QA Manager. In some cases, the authority to review and approve QAPPs is delegated to another 

part of the EPA organization covered by the same QMP. In cases where the authority to review 

and approve QAPPs is delegated in writing by EPA to another organization (e.g., a Federal 

agency or a State under an EPA-approved QMP when the environmental data operation has been 

delegated to that organization for implementation), it is advised that the EPA Project Manager 

and EPA QA Manager should be involved in the review and approval steps. 

Reviewers with expertise in the project specific areas, such as program managers (decision 

makers), QA staff independent of project management, and field and laboratory technical staff, 

should review the plan. These reviewers should:

 ensure that the information is accurate and complete;

 ensure that all appropriate elements are addressed;

 ensure that the plan identifies the project's technical and quality objectives, and that the 

intended measurement and data acquisition methods will satisfy these objectives;

 confirm that the planned assessment procedures will be adequate to evaluate the project; 

and

 confirm that there is a process to identify any limitations on the use of the data.

1.7.3 Approving a QAPP

The EPA organization's QMP establishes how, when, and by whom development, review, 

approval, and effective oversight of QAPPs should occur. This includes processes for external 

(non-EPA) organizations that prepare QAPPs. For EPA projects, the Project Manager (or Project 

Officer in the case of external funding), and the QA Manager usually approve the QAPP. For 

external non-EPA projects, the responsible organization’s Project Manager or Principal 

Investigator and QA Manager may review and approve the QAPP and then submit it for EPA 

approval (unless that EPA organization has specifically delegated approval in its Agencyapproved QMP). It is also beneficial if other key staff such as the laboratory or division directors,

and prime contractors and subcontractors, sign the plan to indicate their review and approval.
QAPP Handbook 13 December 2012 Agency Review Draft

No work involving environmental data collection, production (or generation), or use shall be 

started until the QAPP has been approved by the EPA QA Manager (or authorized 

representative).

Some organizations, according to their QMP, may permit conditional approval of a QAPP in 

situations where only non-critical deficiencies are unresolved (e.g., missing a final organizational 

chart). Conditional approval allows project work to start while the QAPP is finalized. The QAPP 

is then resubmitted for approval when the remaining deficiencies are resolved. Any work 

performed under conditional approval should be in accordance with the fully approved QAPP, 

with any deviations being fully documented for resolution.

1.8 DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP

This section offers guidance on Standards Annex B2.2, QA Implementation and Revision. 

1.8.1 Distributing a QAPP

All applicable personnel involved in the project should retain or have access to the current 

version of the QAPP. This may include the Project Manager, QA Manager, modeler, data 

reviewer, laboratory manager, field team leader, and any essential contractor and subcontractor 

personnel involved with the project.

1.8.2 Implementing a QAPP

The organization performing the work is responsible for ensuring that the QAPP is implemented 

as written and approved, whether this work is conducted by EPA or non-EPA personnel. 

Ultimately the Project Manager is responsible for project activities. A clearly written QAPP will 

help the Project Manager implement the plan, because all project personnel will understand the 

specifications before the start of data production or data use activities. However, effort is needed 

throughout the entire project lifecycle to ensure by direct evidence that the QAPP is being 

implemented properly as planned. 

1.8.3 Modifying a QAPP

The QAPP’s effective period of approval will be defined by the parent EPA organization that 

approves subordinate EPA organizations and non-EPA organization QAPPs. In general, the 

QAPP approval should encompass the project-specific activities and proposed timeframe to 

complete the activities. A QAPP approval for an external agreement may not exceed the period 

of performance for the agreement or no-cost extensions to the period of performance.

The QAPP should be kept current and all appropriate personnel involved in the work should 

have easy access to a current version of the QAPP. Although the approved QAPP must be 

implemented as prescribed, it is not inflexible. Due to the complex and diverse nature of 

environmental data operations, changes to original plans are often needed. When such changes 

occur, the organization’s QA Manager (or designee), together with the Project Manager, 

determines if the change significantly impacts the technical and quality objectives of the project. 

When a substantive change is warranted, the originator of the QAPP modifies the QAPP to 

document the change and submits the revision for approval by the same authorities that 

performed the original review. The EPA Project Manager should determine whether the need to 
QAPP Handbook 14 December 2012 Agency Review Draft

revise the QAPP merits stopping project activities until the revised QAPP is approved (e.g., if 

serious safety threats are identified that haven’t been addressed in the original QAPP), or if the 

type of revisions required do not necessitate stopping work (e.g., the data validation firm named 

in the QAPP went out of business and an equally competent firm was identified to fill that role). 

All revisions should be documented in the QAPP revision history.

For programs or projects of long duration such as multi-year monitoring programs, the QAPP 

should be reviewed regularly as specified in the organization’s QMP. If it is necessary to modify

the QAPP during the timeframe when it is in effect, the same steps should be taken as in the 

initial phase to ensure that the revised QAPP is reviewed and approved by the original approvers 

or by appropriate personnel as specified in the organization’s QMP. 

1.9 PERIOD OF APPLICABILITY

This section offers guidance on Standards Annex B2.3, Applicability of QAPPs.

Consistent with EPA policy, this Handbook is valid for five years from publication and will be 

subject to review at which time it will be revised and reissued, reaffirmed without change, or 

withdrawn. If circumstances warrant, this handbook may also be revised during the course of the 

five-year period.

1.10 SUPERSESSION

This document conforms to the current editions of Quality Policy CIO 2106, and Quality

Procedure 2106-P-01.0, as well as the Internal and External Standards. This document 

supersedes the previous QAPP-related guidance documents EPA QA/G-5 (EPA 2002a) and

EPA QA/G-5M (EPA 2002c).
QAPP Handbook 15 December 2012 Agency Review Draft

CHAPTER 2

QAPP ELEMENTS FOR THE

COLLECTION OF DATA BY DIRECT MEASUREMENT

The collection of data by direct measurement describes the activities where the Agency, its 

contractor, or organization working in agreement with Agency, gathers new data for a project. 

The internal Standard applies to Agency activities, the External Standard to non-EPA 

organizations. The two Standards run in parallel with the exception of Internal Clause 7.11 

Management of Quality Requirements for External Agreements, which applies only to Agency 

personnel involved with organizations having agreements with the Agency.

2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA BY 

DIRECT MEASUREMENT

The QAPP integrates all technical and quality aspects for the life cycle of the project including

planning, implementation, and assessment. The ultimate success of an environmental program or

project depends on the quality of the environmental data collected, produced, and used in 

decision-making, and this quality depends significantly on the adequacy of the QAPP and on its 

effective implementation. Probably the most important part of the QAPP is in the planning for 

obtaining the data or information because an error at this stage may be difficult to rectify once 

the project is underway. One of the key outputs of the systematic planning process is the 

construction of a conceptual model that describes the scientific and engineering process under 

investigation. The conceptual model is an important tool for organizing information about the 

current state of knowledge and understanding of the project, as well as for documenting key

theoretical and practical assumptions underlying the data and information collection. Further 

details on systematic planning are to be found in Annex C of the Standards and EPA 2006a.

The format of the QAPP is not critical except that all required elements must be addressed and 

the content of the QAPP presented in a logical, easily understood manner. In some cases, certain 

elements will not be appropriate for a particular project. Elements that do not apply can be 

addressed with a simple statement of why the information is not relevant or with a crossreference to another approved document in which the information appears. It is acceptable to add 

sections to a QAPP beyond those identified herein. For example, if a project includes the 

temporary care of live animals such as laboratory mice, additional sections relevant to activities 

involving the care and feeding of the animals could be included. In addition, any other applicable 

policies or procedures (e.g., administrative, IT, or long-term data management) may be 

referenced or included as beneficial for the quality of the project. If use of a particular 

information system or hardware/software configuration is vital to project success, the QAPP 

should address IT quality considerations, including a listing of IT-related procedures that will be 

applied. EPA organizations should consider the set of CIO policies, procedures, standards, and 

guidelines for this purpose. For non-EPA organizations, the applicable external agreement may

incorporate relevant IT guidelines.

Documentation, such as an approved Work Plan, SOPs, model calibration reports, etc., may be 

referenced in response to a particular required QAPP element to reduce the size of the QAPP and 

the time required for preparation and review. All referenced documents should be attached as 
QAPP Handbook 16 December 2012 Agency Review Draft

appendices to the QAPP itself or be placed on file with the appropriate EPA organization and 

available for routine referencing when needed. 

This chapter focuses on QAPPs for projects that involve the generation of new data. The

elements are arranged according to the Plan-Do-Check-Act project life cycle. 

2.2 PROJECT MANAGEMENT (PLAN)

Planning, the first part of the construction of a QAPP, deals mainly with the management 

aspects of the project. Most of the structure is simple good management practices, the only QArelated part being the establishment of data or project quality objectives and measurement 

performance criteria (section 2.2.6). The section offers guidance on Standards Clause 7.5, 

QAPP, Standards Clause 7.7, Systematic Planning, Annex B3.3 Project Management, and Annex 

B3.4 Data Acquisition.

The elements in this section address the format and disposition of the QAPP, project 

administrative functions, project information and goals. These elements document the backbone 

of the project planning process and lay the groundwork for the more technical elements. The 

QAPP must describe the project adequately and the elements that can address the basic project 

management and objectives of the work include:

 title, version number, and approval/sign-off sheets;

 document format and table of contents;

 distribution list;

 project organization and schedule;

 project/problem background and description;

 data or project quality objectives and measurement performance criteria; 

 special training requirements/certification; and

 documentation and records requirements.

The records for a QAPP should conform to the EPA Records Management Policy (EPA 2009a).

2.2.1 Title, Version, and Approval/Sign-Off 

Each QAPP should include a page with the title of the project and the name of the organizations 

involved in various aspects of that project. The version and number assigned to the QAPP should 

also be clearly identified along with the title. It is acceptable to create separate title pages and 

signature pages, as long as the document title, version number, and date appear on the signature 

page. The names, titles, signatures, and signature dates of those approving the plan are also 

placed on this page. Individuals responsible for approving the QAPP may include the 

organization’s Technical Project Manager and QA Manager, and the EPA (or other funding

agency) Project Manager and QA Manager. Their signatures indicate both their approval of the 

plan and commitment to follow the procedures noted. Other key personnel that may sign the plan 

are the laboratory directors, the field operations manager, other QA officers, prime contractors, 

and subcontractors.

This approval information is typically the first page of the QAPP. Depending on the 

organization’s administrative policy, QAPP approval could also be in a separate memorandum. 
QAPP Handbook 17 December 2012 Agency Review Draft

The signature dates indicate the earliest date when the environmental data operations portion of 

the project can start (i.e., its effective date). The QA Manager will also determine if digital 

(electronic) signatures are acceptable for the approval of the QAPP.

In addition to the title, version number, and approval signatures, it is important to include a 

revision history. Each time the QAPP is revised, as approved by the QA Manager, the version 

number should be updated and the revision history should be amended to include a brief 

summary of the change and date.

2.2.2 Document Format and Table of Contents

The QAPP should be organized such that it meets the project’s needs, can be reviewed 

efficiently, and meets the document control requirements of the QMS under which it is 

developed. A document control format, such as the example shown in Figure 4, or footer created 

for each page to show revision status may be used to support QAPP development. 

Figure 4. Document control format example

The Table of Contents will generally list QAPP elements, as well as any tables, figures, reference

sections, and appendices necessary to the project. If the QAPP author prefers to organize the plan 

differently than how the elements are organized in this Handbook or in the UFP-QAPP 

worksheets (EPA 2005b), a table may be inserted here to cross-reference where the information 

for each element may be found to simplify review. Standard operating procedures (SOPs) may

be included as appendices to the QAPP. Depending on the type of project, sampling methods, 

analytical research protocols, or data management procedures may be attached. If SOPs or other 

data gathering, data analysis, or evaluation protocols are not documented in, or attached to the 

QAPP, they must be available to the project team and QAPP reviewers through some other 

means. In the case where proprietary standards (for example, those of ANSI or ISO) are used, 

reference to the location on the relevant web-site may be sufficient.

2.2.3 Distribution List

The distribution list identifies all individuals who should get a copy of the approved QAPP, 

either in hard copy or electronic format, as well as any subsequent revisions. Key personnel 

responsible for project implementation and/or funding, and who should have the currently 

approved QAPP, should be listed with their project titles or positions, organization names, email 

addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel 

who will need access to it, the distribution list also serves as an easy reference of who needs to 

be alerted and provided with a revised version of the QAPP in the case that modifications are 

necessary. Some organizations choose to provide the distribution list on the title or approval 

page, others elect to include this list in the project organization section when listing key

personnel and their contact information.

Project Name/# __________________

Section # _______________________

Revision # ______________________

Date ___________________________

Page ___________ of ____________
QAPP Handbook 18 December 2012 Agency Review Draft

2.2.4 Project Organization and Schedule

It is important that roles and responsibilities are well defined prior to initiating project activities. 

Those individuals involved with the major aspects of the project are named in this element along

with their project responsibilities. For example, the people responsible for maintaining, updating, 

and overseeing implementation of the QAPP would be named here. The personnel included in 

this element should include the lead scientists, researchers, modelers, consultants, and contract

and subcontract QA representatives. If the actual personnel cannot be initially identified, then 

the position description of that person’s function should be given. 

An organizational chart or table can be very helpful, and should be included if appropriate. It is 

also helpful to indicate lines of communication among individuals or groups, and this can be 

shown easily on an organizational chart or an organizational network diagram. While a single 

individual may have more than one responsibility in a project, the project should be organized 

such that any person having QA responsibilities is independent of those generating and using 

project information. If this is not possible, owing to the size of the organization or for some other 

reason, an alternative method of ensuring effective QA review should be specified in the QAPP.

The level of detail included in the schedule is left to the discretion of the QAPP authors. It may

be beneficial to have a very detailed and strongly stated schedule for the project to follow. This 

increases the risk of requiring QAPP revisions if the schedule needs to be changed during the 

lifetime of the project unless the QAPP explicitly states it will not be revised simply due to 

schedule delays. It is useful to include critical points in the project such as expected date of 

QAPP approval, sub-section start and end points, dates for sampling, dates by which analyses are 

needed, or dates modeling subroutines need to be completed. When creating the schedule, 

allowance should be made for potential delays, down-times, maintenance off-lines, and general 

inefficiencies inherent in any project. If the project includes regulatory or court-mandated 

deadlines, these should be highlighted to ensure their importance is noted. For projects in which 

milestones or deadlines are not well defined, a more generalized work schedule can be 

formulated.

2.2.5 Project Background, Overview, and Intended Use of Data

This overview should give the reader an understanding of the problem to be solved, along with 

any pertinent background information for the project. It describes why the project will be done 

and what goals the project intends to accomplish. The general project goals stated here form the 

foundation for the entire study. Equally important, the development and documentation of this 

element will help ensure that all project team members clearly understand and agree on the 

underlying purpose of the project, increasing the likelihood that the project design will address 

and accomplish that purpose.

In addition to the general overview, include information on the background and use of previously

approved QAPPs relevant to this project. In addition to information about the project, it is good 

practice to include an outline of information that is currently not sufficiently developed. Clearly

state who needs the information and what the intended use of this information will be. Problems 

that are more complex will lead to more extensive information in this section. The reader should 

be able to understand the importance or context of the project. The general project goals stated in 

this section will be refined in Section 2.2.6.
QAPP Handbook 19 December 2012 Agency Review Draft

2.2.6 Data/Project Quality Objectives and Measurement Performance Criteria

EPA requires the use of systematic planning for all projects that involve the collection or use of 

environmental data, and the QAPP documents the outcomes of this process (see also Annex C of 

the two Standards). EPA encourages the use of the Data Quality Objectives (DQOs), which may

also be called project quality objectives, depending on the organization’s preference. The DQOs 

are established from total study variability as the overall qualitative and quantitative goals of the 

project. If the DQO process (EPA 2006a) is implemented, the QAPP houses the outcomes of that 

process. For example, the project background, the number and type of samples needed, and the 

designs for sample collection and analytical measurement are all included in the QAPP. 

Under any systematic planning process for a project that includes environmental data collection 

it is desirable to define tolerable levels of uncertainty for components of the total study

variability. Total study variability is due to natural (field) variability and measurement 

(laboratory) variability. The effects of field variability may be minimized by selecting an 

appropriate sampling design and number of samples collected (see Appendix B for further 

discussion). Measurement variability can be minimized by careful selection of specific 

measurement techniques (see Appendix C for further discussion). 

DQOs are supported by Data Quality Indicators (DQIs), which are supported by Measurement 

Quality Objectives (MQOs). These DQIs are linked to field and measurement variability (see 

Appendix B for further discussion). MQOs are sometimes known as Measurement Performance 

Criteria (MPCs) depending on the protocols of the organization. See Appendix B for definitions 

and details of DQIs and MQOs. It is essential that the definition of what is meant by MQOs or 

DQOs is made clear early in this section. Some organizations use the same acronyms for 

different operations. For example, some organizations use DQOs in place of DQI performance 

measures, or use MPCs for these measures.

During the DQO or other systematic planning process, DQIs are considered, and specific MQOs 

are set to ensure that data are of appropriate quality for their intended use. These MQOs are the 

criteria against which the data are measured in the “Do” step of the “Plan-Do-Check-Act” cycle. 

MQOs/MPCs are the cornerstones of the technical sections of the QAPP. 

The traditional DQIs considered in project planning include:

 precision;

 bias;

 representativeness;

 comparability;

 completeness; and

 sensitivity.

While historically there has been considerable attention directed to bias, precision, and 

sensitivity, it is really representativeness that is probably the single most important indicator of 

data quality. Representativeness is a qualitative measure of the degree to which data accurately 

and precisely represent a characteristic of a population. 
QAPP Handbook 20 December 2012 Agency Review Draft

A poor sampling design with very high quality analytical laboratory analyses might give data 

that meet MQOs for precision, bias, sensitivity, comparability, and completeness while not 

providing information that is actually representative of the characteristics of the population of

interest. Throughout this guidance, the concept of representativeness (as well as the other DQIs) 

will be used to motivate the need for QA and QC elements. If the data collected fail to be 

representative of what was intended to be sampled, no amount of statistical manipulation can 

make the data set valid. Great attention should be given to ensure that what is sampled or 

collected truly represents the project population of interest. It is axiomatic that a sufficient 

number of samples should be taken in order to achieve sense of representativeness. 

For each DQI there should be an indication of how rigorous the measurement needs to be. 

Without making this clear, data may be inadequate for final use, or conversely, more expensive 

to collect or produce than necessary because a higher standard of quality is targeted than is 

required for the project purposes.

The QAPP should indicate how rigorous the collected data or information should be to answer 

the question, resolve the problem, or support the decision to the level desired and needed for the

project. For projects that include environmental data collection or production, acceptance or 

performance criteria information should be determined during the project planning process and 

documented in the QAPP. 

For example, in an enforcement decision, this discussion could focus on regulatory or action 

levels and the quality of the data required to make and enforce decisions in relation to the project 

action limits. Some organizations, especially those using the UFP-QAPP, distinguish between 

screening data (data of known quality suitable for use for interim decisions) and definitive data 

(data of known quality suitable for use for final decisions). Whether for screening data, definitive 

data, interim data, or count data, the same tenet holds true: it is important to understand the 

intended use of the data and to set and meet measurement performance criteria that ensure the 

quality of that data are sufficient to achieve the decision objectives.

It is important to note that the selection of DQOs and MQOs must be grounded in reality. An 

arbitrary selection of these numerical values, or default to instrument performance 

characteristics, will rarely lead to the generation of data quality sufficient to meet the needs of 

the project. The MQOs must be clearly linked to the DQOs, with these in turn linked to the 

project’s overall objectives.

In addition to planning for the type of data needed to address a particular project, it is important 

to also consider the way those data will be analyzed. That is, what statistical or other evaluation 

methods will be applied to the data in order to reach a conclusion for the project? The QAPP 

should address all aspects of project planning from the problem statement, through data and 

information needs, through quality criteria to be applied, through laboratory analyses of samples, 

through statistical analyses of the data, through the final project decisions and reports, to 

disposition and maintenance of the data and records. 

Establishing DQOs and MQOs for exploratory projects (for example, research studies) where 

little or no information on the quantitative magnitude of what is expected is difficult. The use of 

information from similar or related studies is useful.
QAPP Handbook 21 December 2012 Agency Review Draft

For example, consider an experiment designed to investigate the neutralizing properties of a 

chemical on a known contaminant mixture. For this experiment, it may be sufficient to use 

qualified statements such as “the precision (MQO) for measuring reduction of acid is expected to 

be of the order +/- 10% based on similar experiments as documented in XXXX”. A discussion 

documented in the QAPP of the consequences (or results) of achieving different realized DQOs 

or MQOs is advisable.

In some circumstances, DQOs may be stated in qualitative terms having definitive reference 

points. For example, “The project will produce data that will qualify to receive the ‘A’ rating 

with respect to the rating system described in Section 4.4.2 of the Procedures for Preparing 

Emission Factor Documents (EPA-454/R-95-015)” (EPA 1997a). Although it is stated in 

qualitative terms, this DQO is measurable using specific acceptance criteria that are presented in 

the cited document. Simplistic descriptors such as “very good” or “acceptable for this project” 

are to be discouraged unless these are specifically defined in the QAPP.

2.2.7 Special Training Requirements and Certification

Special training or certifications are sometimes necessary for project personnel and laboratories 

associated with projects. This may include such things as having project personnel complete 

specialized hazardous material handling training, being skilled in the collection of samples for 

trace metal analysis, being trained in global positioning technology, or being trained in specific 

sampling techniques. Similarly, project personnel may require special security clearances such as 

clearance to enter a military site. Laboratory certification for the analysis of certain types of 

samples may also be necessary. The QAPP should describe how the project will ensure that 

competent personnel are available to perform the work. A method for ensuring competency helps 

ensure project specific requirements are met. For example, the ORD Policies and Procedures 

Manual, Section 13.4, Minimum QA/QC Practices for ORD Laboratories (EPA 2001b), states 

“Prior to performing sample analysis with a new method for which proficiency has not been 

previously demonstrated, the analyst must demonstrate proficiency with the method by

completing the following: (1) perform valid initial calibrations, (2) perform method detection 

limit determination, (3) demonstrate that they can meet all minimum QA/QC acceptance criteria 

as presented in the method document, e.g. the SOP, and (4) if available, satisfactorily analyze a 

performance evaluation sample or a second source standard”. The QAPP should also specify

how this information will be documented, where the records will be kept, and indicate who is 

responsible for ensuring that these special training and certificate needs are met and documented. 

2.2.8 Documentation and Records Requirements

The QAPP should describe the process and responsibilities for ensuring that project personnel 

will receive the most recently approved project documents such as the QAPP, SOPs, and other 

documents used throughout the project operation. The QAPP authors may choose to tell how 

these documents will be updated and how information regarding updates will be communicated. 

The QAPP should identify the information to be included in the project data files (or project 

record) and its format. This might include the following items:

 sampling collection and handling records such as

o field notebooks or operational records,
QAPP Handbook 22 December 2012 Agency Review Draft

o Global Positioning System (GPS) data,

o chain-of-custody forms, and

o sample receipt records, including sample tags and shipping bills;

 analytical records such as

o analytical log books,

o test method raw data and QC sample records,

o Standard Reference Material and/or proficiency test sample data, and

o instrument, equipment, and model calibration information; and

 data assessment records such as

o validation of software used,

o input and output files as results of code,

o statistical methodologies, and

o data-base test procedures.

In some cases, access and the final disposition of records may be regulated or required by policy

and this status should be included in the QAPP. The QAPP should clearly indicate where all 

project documents, reports, records (including digital records) will be stored and for how long. 

2.3 DATA ACQUISITION (DO)

This is the heart of a QAPP as it documents the requirements for sample identification (where 

the samples will be taken), sample collection (how to preserve the integrity of each sample), and 

sample handling (the management of the samples). This section offers advice on the Standards 

Clause 2.1, Scope, Clause 7.7, Systematic Planning, and Annex B3.4, Data Acquisition. 

The elements in this section address all aspects of data production and collection to ensure that 

appropriate methods for sampling, measurement and analysis, data production, data handling, 

and QC activities are employed and documented. These elements describe the actual methods or

methodology to be used for the collection, handling, analyses of samples, and the management 

such as compiling and handling of the data. Where applicable, the elements shall be addressed in 

the QAPP and include:

 sampling collection design, experimental design, and sampling tasks;

 sampling procedures and requirements;

 sample handling and custody requirements;

 analytical methods requirements;

 quality control requirements;

 instrument/equipment testing, calibration and maintenance requirements, supplies and 

consumables; and

 data management requirements.

The QAPP should provide detailed information on the methods to be used. It is important to note 

that if the designated methods are well documented and are readily available to all project 

participants, citations are adequate. 
QAPP Handbook 23 December 2012 Agency Review Draft

When procedures or methods are not commonly used or readily available, detailed copies of the 

methods and SOPs should accompany the QAPP either in the text or as attachments.

2.3.1 Data Collection Procedure, Experimental Design, and Sampling Tasks

This element describes the project’s data collection or research experimental design. Keys to this 

element are the assumptions made and how the data will be obtained. This element explains the 

“how and why” of the project’s information collection design to ensure that the appropriate data 

are collected for this project. Input for this element will come from the project’s systematic 

planning (e.g., the DQO process) and also see Annex C of the two Standards. 

For data collection there are two classes of sampling designs to consider: probability-based and 

judgmental, and they have very different properties. The former are sometimes called statistical 

designs, and the latter sometimes called directed sampling designs. Strong statistical conclusions 

are available with probability-based designs but not with judgmental designs. Use of professional 

expertise and/or historical knowledge about the site can improve development of statistical or

judgmental sampling designs. 

Key questions to be considered include:

 is this project to be comparable with previous sampling or analytical efforts, or with a 

health-based or regulation standard?

 can samples or measurements be taken according to a probability-based design?

 is the objective of the sample to estimate an average or to find a hot spot?

 is there a reference or background population that can be used as a comparison to the 

target population?

 will sampling units be chosen in advance, in the field based on visual evidence, or using 

other technical evidence?

 is there a network of sampling sites that will be visited periodically or where sampling 

will be performed continuously?

 do all the samples need to be taken simultaneously?

 is the target population approximately homogeneous or is it heterogeneous in nature 

needing stratification or division into approximately homogeneous areas?

 can samples be composited? and

 are there any potentially important sources of variability which affect the sampling such 

as tidal cycles, seasonal differences, and rain and wind patterns?

The answers to these questions should have been considered during the planning process and 

help to determine allocation of resources for obtaining samples for analysis. 

For example, when the project involves a physical site, this element of the QAPP may involve:

 defining the size of the area, shape, volume, or time that is to be represented by a sample 

(called the scale of representativeness) as part of the justification for how the sampling 

sites and durations will be selected; 
QAPP Handbook 24 December 2012 Agency Review Draft

 specifying the type and number of samples to be collected, and their locations; and 

 providing detailed schedules for sampling and analytical activities, test runs, and reviews.

If, instead, the project is to develop a new analytical measurement method, this step could 

involve:

 defining the procedure to be used for the testing; 

 specifying the levels at which testing will occur;

 setting a timescale for sequential instrument readings; and 

 clearly stating any extraneous variables (e.g., temperature) that will be controlled.

The QAPP should describe the experimental data production or data collection design for the 

project, including as appropriate:

 the sampling design (e.g., systematic grid if spatial, time-interval if temporal); 

 the rationale for the design; 

 the types and numbers of samples required;

 restrictions due to resource constraints;

 need to collect background samples;

 the sampling locations and frequencies; 

 rationale for ensuring representativeness of data; and

 measurement parameters of interest.

Advice on selecting the appropriate sampling design may be found in Chapter 2 of the Guidance 

for Choosing a Sampling Design for Environmental Data Collection, EPA QA/G-5S (EPA 

2002b). Advice on experimental data collection is available from the standard statistical 

literature under the title “Design of Experiments” or “Experimental Design”.

2.3.2 Sampling Procedures and Requirements

Standardized sampling procedures provide consistency between samplers; facilitate collection of 

accurate, precise, and representative samples; and help to ensure data comparability and 

usability. These may be in the form of SOPs established by the organization or may be unique to 

the investigation in hand. The QAPP should provide a list or table of all field sampling 

procedures and brief descriptions of sampling procedures. In many instances reference to where 

the SOPs are located should be sufficient but in some cases may be attached to the QAPP to 

allow for review and approval. Any SOPs that are modified to meet project-specific needs should 

be attached to the QAPP.

Sampling procedures should include SOPs, when available, for sampling each matrix and each 

analytical parameter for each type of equipment and technique. The SOPs provide detail on the 

appropriate number, size, and type of sample containers to be used for collection of each field 

sample and field QC sample and the proper temperature, light, and chemical preservation 

procedure for those samples.
QAPP Handbook 25 December 2012 Agency Review Draft

Also included should be information on the techniques to be used for specific locations due to 

difficulties in sample collection. These can include restrictions on physical sample collection

(e.g., inability to collect the sample where planned), availability of trained personnel (e.g., lack 

of a certified technician), or seasonal influences on the obtaining of a physical sample (e.g., 

adverse winter conditions).

The QAPP should describe how samples will be collected. The selected sample collection 

procedures should be appropriate to ensure that project personnel collect representative samples 

in a consistent manner for all required sample matrices and locations; that contamination is not 

introduced during collection; and that sample volumes are properly preserved in order to meet 

project objectives.

The QAPP should include a description of preservation procedures (temperature, light, or 

chemical) that maintain sample integrity in the field, prior to and during shipment to, and 

immediately upon receipt by the off-site or mobile on-site laboratory. The QAPP should 

document requirements for sample volumes, container types, number of containers, and 

preservation procedures for each analytical group, matrix, and concentration level.

The QAPP should provide details on the procedures for both the initial cleaning of sampling 

equipment and also subsequent decontamination procedures that will be followed during the 

sampling event. These procedures will help ensure that collected samples are representative of 

the sampling location by verifying that sampling equipment is clean and free of target analytes, 

Chemicals of Concern, or interferences. Cleaning and decontamination procedures should cover 

all equipment that contacts a sample. If the sampling equipment is disposable (“one use only”), 

procedures for cleaning and decontamination are not necessary; however, the QAPP should state 

that disposable equipment will be used and where that equipment will be deposited.

Development of a standardized table customized to the needs of the organization is encouraged 

as it leads rapid review and ease in tracking adherence to sample analyses requirements.

2.3.3 Sample Handling, Custody Procedures, and Documentation

The QAPP should include all sample collection documentation and sample handling, tracking, 

and custody procedures needed to ensure that sample integrity and custody are maintained. 

Without sample integrity, and without clarity regarding the chain-of-custody for all samples, it 

may be impossible to assert that the data are of adequate quality to meet project objectives. The 

procedures should address sample collection, packaging, handling, and shipping, as well as 

records, receipt of laboratory samples, archiving, and disposal. These may be included by

reference if an approved SOP is maintained by the organization.

It is important to note that the detail needed for handling, chain-of-custody, and documentation 

should be commensurate with the goals or objectives of the project (application of the Graded 

Approach). There are many sample collection procedures that do not need extensive procedures 

(for example, use of a sample inventory, or use of an EPA number to preserve some degree of 

confidentiality). Reference to the organization’s Quality Management Plan and the 

organization’s QA Manager should clarify what is needed to meet the project’s objectives.
QAPP Handbook 26 December 2012 Agency Review Draft

Sample Handling and Tracking

Proper sample tracking systems support the chain-of-custody procedures, which help to ensure 

sample authenticity and data defensibility. The QAPP should document the procedures that will 

be followed to identify and track samples that are collected in the field, analyzed on-site, and 

delivered or shipped to an off-site laboratory for analysis, as well as samples transferred 

throughout the laboratory. If samples are shipped to an off-site laboratory, then the laboratory’s 

sample handling and tracking system should also be described.

The sample handling and tracking procedures should include the following descriptions:

 the sample numbering system for field sample collection with an example (if applicable, 

the numbering system should follow specific programmatic requirements that apply to 

the project. A systematic approach for numbering samples should be used so that each 

sampling location, matrix type, sample depth or height, and date and time of collection 

can be uniquely identified and cross-referenced to the programmatic sample number);

 the sample container identification information;

 the latitude, longitude, and elevation (altitude) for the sample collection location;

 the laboratory sample tracking procedures (if laboratory identification numbers will be 

used to track samples internally, the laboratory procedure must describe how these 

laboratory identification numbers will be cross-referenced with the sample number 

assigned in the field); and

 the sample storage procedures used by the off-site or mobile on-site laboratory.

To demonstrate the project’s sample handling process, the QAPP should include a table that 

shows the flow of samples from the time of collection to laboratory delivery to final sample 

disposal. The table should identify each component of the project-specific sample handling 

system; indicate the personnel (and their organizational affiliations) who are primarily 

responsible for ensuring proper sample handling, custody, storage, and disposal; and specify the 

length of time that samples, digestates and extracts, and biological collections will be retained by

the laboratory prior to disposal.

The QAPP should also describe how samples will be delivered or shipped to the laboratory. The 

description should include the name of the carrier service, if applicable, and define how samples 

will be batched or grouped when sent to the laboratory. The QAPP should include provisions for 

packaging, marking and labeling, and shipping samples in compliance with the Department of 

Transportation regulations for shipping hazardous and nonhazardous materials (see also 49 CFR 

171). Air carriers that transport hazardous materials require compliance with the current edition 

of the International Air Transport Association Dangerous Goods Regulations.

Shipment papers, including bills of lading and airbills, should be retained by the laboratory with 

chain-of-custody records. Examples of all sample shipment forms to be used should be attached 

as an appendix to the QAPP.
QAPP Handbook 27 December 2012 Agency Review Draft

Sample Custody Procedures

A sample is in “custody” if it is in the actual physical possession of authorized personnel or in a 

secured area that is restricted to authorized personnel. For some projects, an evidentiary paper 

trail documenting sample custody is required in order to meet project quality objectives. Since it 

is often difficult to predict what samples or projects will require proof of custody after the fact, 

all data collection events should employ documented chain-of-custody procedures to ensure data 

authenticity and defensibility. The evidentiary trail from sample collection through data 

production and archiving is maintained using sample custody procedures and documented by

complete chain-of-custody records. Chain-of-custody procedures ensure accountability for the 

location and integrity of the sample at all times. The ASTM document D4840-99 Standard 

Guide for Sampling Chain-of-Custody Procedures contains useful information regarding chainof-custody procedures (ASTM 1999).

The QAPP should describe (or have attached) the procedures that will be used to maintain 

sample custody and integrity and to document the implementation of chain-of-custody 

procedures. Sample custody procedures should include the field sampling team’s procedures for 

maintaining and documenting sample custody from the time samples are collected in the field 

through packaging, shipment, and delivery to the laboratory. Field sampling documents that 

describe chain-of-custody procedures, including SOPs, should be attached to or referenced in the 

QAPP. The laboratory’s procedures for maintaining and documenting sample custody from the 

time the samples are received at the laboratory through archiving and disposal should also be 

attached to or referenced in the QAPP. The use of identification or tracking software such as 

FORMS II Lite TM, should be documented, if applicable. Examples of all chain-of-custody 

documents that will be used during the project should be provided in the QAPP, including chainof-custody forms, sample identification, custody seals, laboratory sample receipt forms, and

laboratory sample transfer forms.

Sample Documentation

The QAPP should describe sample documentation procedures that will be followed for the 

project. Proper documentation of field sampling activities helps ensure sample authenticity (i.e., 

the sample identity is correct) and data integrity. The QAPP should also indicate what 

information should be marked on the sample collection container label (e.g. time, date, sample

number), the analyses to be performed, preservatives used, and name (with signature) of the field 

technician completing the sample collection.

Documentation for sample collection includes, descriptors of sample type (duplicate, QC check, 

etc.), field logbooks, field data collection forms, field calibration forms, chain of custody, and 

sample container identification. The QAPP should specify the required sample identification 

information and include an example. An electronic system, such as a bar code or FORMS II Lite 

TM (which retrieves information stored elsewhere), may be used. The QAPP should also describe 

how the information on the label will be preserved (e.g., by covering the label with clear tape to 

minimize water damage during transit).
QAPP Handbook 28 December 2012 Agency Review Draft

2.3.4 Analytical Methods Requirements and Task Description

The QAPP should identify the analytical methods, sensitivity, precision, bias and acceptance 

limits for QC results, equipment required, sub-sampling or extraction methods, laboratory

decontamination procedures and materials (such as in the case of hazardous or radioactive 

samples), waste disposal requirements (if any), and any specific performance requirements for 

the method. Where appropriate, analytical methods may be identified using their number, date, 

and regulatory citation. If data from laboratory analyses do not form part of the overall project 

description, then a simple outline of the data production method is necessary. For example, to 

determine if there have been observable ecological deficits at a study site, plant cover and a 

diversity study should be included in the project summary. The QAPP should then describe the 

number and size of the study sites, the procedure for counting the species (e.g., number of unique 

species or number of specimens of each species), and the procedure for locating the sites.

The QAPP should specify any method performance standards. If a method allows the user to 

select from various options, then the rationale for selection should be given. For non-standard 

method applications such as for unusual sample matrices (e.g., explosives testing of sea 

cucumber tissue), appropriate method performance study information is needed to confirm the 

performance of the method for the particular matrix. If previous performance studies are not 

available, they should be developed during the project and included as part of the project results. 

The organizations or laboratories that will provide the analytical services (all on-site and off-site 

laboratory analytical work, including all prime laboratories, subcontractor laboratories, and 

backup laboratories) should be identified together with the measures taken to ensure laboratory

proficiency. It is recommended that a table of analytical services be included in the QAPP, 

containing the following information at a minimum:

 matrix;

 analytical group or class of compounds;

 concentration level;

 sample location or identity number;

 analytical SOP to the analysis requested;

 preservation and holding time;

 number and type of sample containers;

 required data package turnaround time;

 the number of QA and QC samples needed;

 laboratory/organization (name, address, contact person, and phone number); and

 the arrangements made with a backup laboratory/organization.

If these method requirements are already described in the laboratory’s set of SOPs, these should

be attached or referenced as appropriate. In addition, the format of the data package (electronic, 

hard-copy, etc.) should be identified. These requirements should be commensurate with those 

requirements contained in the overall agreement between the organization and laboratory, 
QAPP Handbook 29 December 2012 Agency Review Draft

including the timeframe for delivery of data and information. It is possible the conditions of the 

agreement will affect the MQOs established earlier and resolution of the differences will be 

needed (for example, the detection limit MQO set at 0.1ppb, but the agreement reporting limit set 

at 0.5ppb).

The measures taken to ensure the laboratory will meet the proposed data quality needs of the 

project should be discussed. When necessary, as determined by application of the graded 

approach, the review of SOPs and the results from performance evaluations using similar 

contaminants of concern in matrices similar to those expected may be needed.

It is important to address what to do when a failure in the analytical system occurs, who is 

responsible for corrective action, and how the effectiveness of the corrective action shall be 

determined and documented. 

2.3.5 Quality Control Requirements

Quality Control (QC) has two important aspects: preventive and corrective. The preventive 

aspect is designed to prevent quality problems before they are observed. This is achieved 

through the use of proactive procedures that minimize variability and bias at each stage of 

measurement. Examples of proactive QC include the use of check sheets, control charts, and 

standardized data recording forms to promote and document consistency in performance of 

specific activities. The corrective aspects of QC can sometimes alleviate situations where there is 

the potential for an error in interpretation. Some of the retrospective activities of data verification 

and validation may be considered examples of corrective action, whereas the use of statistical 

(Shewhart) control charts is an example of proactive corrective activities.

The QAPP should identify QC activities needed for each sampling, analysis, or measurement 

technique. For each QC activity, the associated method or procedure, acceptance criteria, and 

corrective action should be listed. As standard methods are often vague or incomplete in 

specifying QC requirements, simply relying on the cited method to provide this information is 

usually insufficient. QC activities for the field and the laboratory include, but are not limited to, 

the use of blanks, duplicates, matrix spikes, laboratory control samples, surrogates, or second 

column confirmation. A list of commonly used QC samples is included in Appendix C. 

In environmental studies, measurement is the process of obtaining a quantitative value describing 

a chemical or physical property of an individual sampling unit or specimen collected from this 

unit. Measurement may involve direct field measurements using survey instruments or collection 

and handling of physical samples followed by analysis in a fixed or mobile laboratory. Choices 

that the technical team makes regarding sample acquisition, sample handling, preparation, and 

analysis can influence the quality attributes of the resulting data. Sources of measurement 

variability include: 

 within-unit, small-scale variability (influenced by the nature and distribution of the 

characteristic of interest within the sampling unit and the media assayed); 

 physical sample acquisition protocol (sample collection tools and procedures such as 

compositing);

 sample handling and transport; 
QAPP Handbook 30 December 2012 Agency Review Draft

 variability associated with the measurement assay (influenced by the capabilities and 

management of the analytical resources);

 inaccurate determination of strata boundaries;

 inadvertent bias by the sample selector (identification uncertainty);

 homogenization and sub-sampling procedures; 

 sample preparation, extraction procedures, and sub-sampling extract for analysis; and 

 analytical determination. 

Each MQO should have at least one QC procedure designed to prevent unsatisfactory results for 

as many sources of variability as possible. An example would be the proper adherence to 

documented SOPs (a preventative measure) combined with use of Shewhart QC charts 

(statistical process control) to monitor analytical results (a corrective measure).

The QAPP should describe, reference, or attach the procedures to be used to calculate applicable 

statistics such as precision and bias. Copies of the formulae are acceptable as long as the 

accompanying narrative or explanation specifies clearly how the calculations will address 

potentially difficult situations such as missing data values, “less than” or “greater than” values, 

“below detection”, “not detected” and other common data qualifiers.

One use of routine QC results is assessment of the laboratory's internal operations; for example, 

the routine analysis of calibration blanks to test for inorganic contamination or a problem in 

system calibration. In this case, a DQI could be just the actual blank result, expressed in 

appropriate units, with an MQO equal to the laboratory’s internal acceptance criterion. A poor 

result from a routine calibration blank should result in an immediate corrective action by the 

laboratory. This action may be limited to simply rerunning a calibration blank to demonstrate 

that the first result was an aberration. While it is important to verify that the laboratory is 

monitoring and controlling the quality of their internal operations, results from samples like a 

calibration blank are less important in project planning from the DQO perspective than other 

potential sources of error that may not be controlled. The same can be said for overall analytical 

calibrations. A properly calibrated analytical system is always required and expected for analysis 

of project samples. 

2.3.6 Instrument/Equipment Testing, Calibration and Maintenance Requirements, QA/QC for 

Supplies and Consumables

It is necessary to recognize the growing importance of field instrumentation and use, and so the 

same care and attention given laboratory equipment should be given to field equipment. If the 

field equipment is intended to give less precise readings, lesser QA/QC requirements may be 

needed as documented in the organization’s QMP.

Many of these requirements are part of a laboratory’s quality system and should be documented 

in the laboratory’s quality manual (however named). It may be sufficient to simply reference this 

quality manual (including revision number and date) or reference the contract arrangement 

between organization and laboratory. If the project has specific requirements different from the 

laboratory’s quality manual, specificity in activities, actions, analyses, and responsibilities for 

their successful implementation may be necessary. The depth of specificity should be linked to 
QAPP Handbook 31 December 2012 Agency Review Draft

the project’s objective by application of the graded approach or consultation with the 

organization’s Quality Manager.

Instrument/Equipment Testing and Inspection

The QAPP should describe how inspections and acceptance testing of instruments, equipment, 

and their components affecting quality will be performed and documented to assure their 

intended use as specified. The QAPP should identify and discuss the procedure by which final 

acceptance will be performed by independent personnel (e.g., personnel other than those 

performing the work) and/or by the EPA project manager. The QAPP should describe how nonconformances or deficiencies are to be resolved, when re-inspection will be performed, and how 

the effectiveness of the corrective action will be determined and documented.

Instrument Calibration and Maintenance

The QAPP should identify all tools, gauges, instruments, and other sampling, measurement and 

test equipment used for data collection or production activities affecting quality that should be 

controlled and calibrated at specified periods, to maintain performance within specified limits. 

The QAPP should include detailed descriptions or references to SOPs describing how calibration 

will be conducted, the frequency of calibration, the procedures for analyzing calibration data, the 

acceptance criteria for calibration limits (with references), what certified equipment is needed, 

how valid traceability to established performance standards will be demonstrated, and how long 

calibration records will be maintained. If no such established standards exist, the basis for the 

calibration should be documented.

The principal reason for performing calibration is to reduce bias to a minimum. The decision as 

to the type of standard the instrument should be calibrated against depends on the overall 

accuracy expected from the instrument, the project objectives (DQOs or MQOs as appropriate), 

and the resources allocated for this part of the project’s development. The DQOs/MQOs of the 

project may be badly compromised should an inadequate calibration be performed. Ineffective 

calibration will often give the impression of precision which can be substantially overwhelmed 

by an increase in bias. See Appendix C for a further discussion on bias.

The QAPP should describe or reference the process for how periodic preventive and corrective 

maintenance of measurement and test equipment or other systems and their components affecting 

quality shall be performed to ensure availability and satisfactory performance of the systems. 

The QAPP should identify the equipment or systems requiring periodic maintenance. The QAPP 

should also discuss how the availability of critical spare parts, identified in the operating 

guidance and/or design specifications of the systems, will be assured and maintained.

Supplies and Consumables

Not all projects will need supplies and consumables considered “critical” to the project. For 

those that do, this element documents the system used to ensure appropriate materials are 

obtained in a timely fashion.

The QAPP should describe how and by whom supplies and consumables (e.g., standard materials 

and solutions, sample glassware/containers, calibration gases, reagents, hoses, deionized water, 

potable water, and electronic data storage media) shall be inspected and accepted for use in the 
QAPP Handbook 32 December 2012 Agency Review Draft

project. The QAPP should state acceptance criteria for such supplies and consumables and have

criteria for disposable sampling equipment and supplies. Where applicable, information enabling 

the needed materials to be located and obtained should be listed in the QAPP, including:

 supply source or vendor;

 procedures for identifying, tracking, storing, and retrieving these materials;

 identification of those responsible for maintaining supplies; and

 any acceptance criteria needed (for example, certificate of testing).

2.3.7 Data Management Requirements

The QAPP should describe the project data management process, tracing the path of the data 

from their production to their final use or storage (e.g., the field, the office, or the laboratory), 

which is very important for ensuring data integrity. The QAPP should describe or reference the 

standard record-keeping procedures, document control system, and the approach used for data 

storage and retrieval of electronic media. The QAPP should discuss the control mechanism for 

detecting and correcting errors and for preventing loss of data during data reduction, data 

reporting, and data entry to forms, reports, and databases. Examples of any forms or checklists to 

be used should be provided.

The QAPP should identify and describe all data handling equipment and procedures to process, 

compile, and analyze the data, including any required computer hardware and software, and 

address any specific performance requirements for the hardware/software configuration used. A 

description of the procedures that will be followed to demonstrate acceptability of the required 

hardware/software configuration should be included (for example, after upgrading software, the 

data should be checked to ensure the migration has occurred successfully). 

Finally, the QAPP should discuss, as necessary, the archiving, data migration, and software 

update changes checks that will be used to ensure no data loss. This includes backup procedures 

for any data stored electronically, and protocols for access to, retrieval from, and photocopying 

of, information archives. In some cases retention, access to, and final disposition of some records 

may be regulated and require field and laboratory QC results to be captured in electronic data 

management systems. In those cases, this element should address and comply with all relevant 

regulations. In other cases, requirements for this element may be stated in the terms and 

conditions of external agreements and the QAPP should identify any requirements set by statute 

or policy.

Further assistance on aspects of data management may be found in Brilis et. al., 2004.

2.4 ASSESSMENTS (CHECK)

The third part of the Plan-Do-Check-Act paradigm considers the technical aspects of QA that 

apply to ensuring the integrity of the data. Not all of these different assessments are necessary 

and should be selected by application of the graded approach that selects the level of assessment 

commensurate with the intended use of the data. The advice given here is drawn from the 

Standards Clause 7.8, Assessment of Data and Information, and Annex B3.5 Assessments.

These QAPP elements ensure that planned project activities are implemented as described in the 

QAPP and that reports are provided to apprise management of the project status and any QA 
QAPP Handbook 33 December 2012 Agency Review Draft

issues that arise during implementation (see also Section 2.5.5). Assessment activities help to 

ensure that the resultant data quality is adequate for its intended use, and that appropriate 

responses are in place to address non-conformances and deviations from the QAPP. Several 

types of assessments that could be included are:

 technical systems assessments;

 performance audits of measurement and analytical systems;

 surveillance operations;

 audits of data quality;

 qualitative and quantitative comparisons to acceptance and other criteria;

 interim assessments of data quality;

 evaluation of unconventional measurements; and

 evaluation of unconventional monitoring projects.

If deviations from the QAPP are identified through a formal assessment or audit, corrective 

actions may be necessary; these are discussed in greater detail in Chapter 3, Section 3.5 of 

Guidance on Technical Audits and Related Assessments for Environmental Data Operations, 

EPA QA/G-7 (EPA 2000). Deviations from the QAPP identified through formal or informal

review may also require corrective actions but, using the graded approach, the project manager 

and QA Manager should confer to ascertain the possible impact on the validity of the data. Their 

joint decision should be documented in the revised QAPP. In some cases deviations from the 

QAPP can be identified by project personnel without the benefit of formal, scheduled 

assessments. This section addresses those situations and describes the process by which the need 

for corrective action is documented, reported, and implemented and its effectiveness assessed. 

2.4.1 Technical Systems Assessments

Technical Systems Assessments (TSAs) can have two forms: field-based and laboratory-based. 

A field TSA includes an auditor monitoring of sampling activities to ensure that the QAPP and 

any field sampling and sample handling SOPs are being followed. Additionally, the field TSA 

should evaluate reporting, handling, tracking, management, and review of data generated in the 

field (e.g. pH, salinity, conductivity, test kits, and GPS data). 

A field TSA may be conducted independently of the laboratory TSA and may even use different 

personnel (auditors). A field TSA can be performed at the start of field sampling activities as a 

readiness review. The readiness review is a technical assessment that is planned and performed 

prior to the initiation of a project to verify that project management has brought the facility and 

applicable measurement systems to a state of readiness to begin the project. 

A laboratory TSA is a thorough assessment of laboratory analytical procedures during which the 

facility (or mobile laboratory), equipment, instrumentation, supplies, personnel, training, 

analytical methods and procedures, laboratory procedures, sample handling and tracking, data 

reporting, data handling and management, data tracking and control, and chain-of-custody

procedures are checked for conformance with the QAPP. 
QAPP Handbook 34 December 2012 Agency Review Draft

There are several ways of conducting a laboratory TSA. The technique of tracing the project 

samples through a laboratory can be used effectively to gain an understanding of the overall 

analysis process during a TSA. If required by the QAPP, samples should arrive with chain-ofcustody papers or appropriate documentation and should be checked for discrepancies needing 

resolution before further analytical work is done. Typical sample handling steps may be 

digestion, extraction, and splitting or making an aliquot of the extract for various analyses. One 

way to trace an assessment trail is to physically follow the trail of the samples through the 

facility, looking in the records for their fate at each step of the trail.

An laboratory TSA can be performed prior to, at the start of, or at any time during field-sampling 

activities. A TSA performed during the data collection phase of a project may be particularly 

helpful in discovering operational problems early that could adversely impact the quality of the 

project results and the usability of the data.

The decision to conduct a field or laboratory TSA, and the intensity of the TSA, should be linked 

to the project’s objectives and intended use of the data. It is important that the principle of the 

graded approach be considered and the extent of the TSA tailored to avoid unnecessary

expenditure of resources.

2.4.2 Performance Audits of Measurement and Analytical Systems

This has two components: proficiency evaluation, and split sample analysis.

Proficiency Evaluation

A performance evaluation (PE) sample is a quantitative assessment in which analytical results 

are generated by a measurement system for a sample that originates outside of a project. PEs 

may also be known as proficiency testing (PT) samples. Statistical analysis of PE sample results 

provides information on routine laboratory performance and the overall accuracy and bias of the 

analytical method. 

Ideally, a PE sample mimics routine field samples in all possible aspects, including similarity of 

container, label, and shipping, such that the analyst is unaware that it is not a routine project 

sample. When possible, PE samples should be of similar matrices and analytes, in the same 

approximate concentrations as expected in the field samples, so that the analyst does not have to 

treat the PE sample differently than typical project samples (e.g., does not have to dilute the PE 

sample if project samples don't have to be diluted). Sample matrix similarity is especially 

important for projects involving solids, sediments, biosolids, and tissues. One potential drawback 

to PE samples is the difficulty in mimicking the complexity of the matrix containing a chemical 

of concern. If the PE matrix is different from the actual sample matrix, difficulties in 

interpretation of readings can occur. In the context of a Quality Program, a PE is used to 

determine if a measurement system’s results are within data quality goals specified in the QAPP. 

PE results are often used to estimate the degree of bias in the measurement system. Although a 

PE sample can identify a problem quantitatively, it typically cannot be used to directly determine 

the cause of the problem. The appropriate choices for the PE samples’ composition should be 

specified in the QAPP. 
QAPP Handbook 35 December 2012 Agency Review Draft

When needed, the QAPP should address the selection of appropriate PE samples. Factors to 

consider include:

 analyte selection; 

 whether PE samples are single- or double-blind, native or synthetic matrix, spiked or 

natively contaminated or both; 

 multiple matrices and concentrations; 

 total number of PE samples; 

 acceptance criteria for PE samples; and 

 analytical methods.

A plan for selection, implementation, and reporting of PE samples should be included in the 

QAPP.

Split Sample Analysis 

One use of split sample analysis is for comparison studies conducted to assess inter-laboratory

precision and accuracy. The sampler collects one field sample and then physically splits it into 

two representative sample aliquots. The samples are then sent to different laboratories for 

analysis. Split samples quantitatively assess the measurement error introduced by the 

organization’s sample shipment and analysis system. Split sample comparability criteria must be 

generated prior to sample collection and documented in the QAPP.

Another use for split sample analysis is for projects involving enforcement actions or compliance 

with a set standard. Split samples are collected and one sample is analyzed by the EPA, the other 

is provided to a party representing the site or facility owner. Here the two samples are 

independently analyzed although EPA may not know the results of the other sample (including 

QC results) until much later in the project’s development. 

For further information on the use of audits see Guidance on Technical Audits and Related 

Assessments for Environmental Data Operation, EPA QA/G-7, (EPA 2000). For further 

information on the use of various QC samples (including split samples), see Appendix C of this 

guidance.

2.4.3 Surveillance of Operations

Surveillance is the observation of ongoing work for the purpose of verifying and documenting

conformance with specified requirements and/or procedures such as those given in the QAPP 

and/or SOP. Surveillance is focused on a particular technical activity, rather than on the entire 

measurement system. It is typically less formal than other types of assessments, but it should also 

include appropriate preparation, conduct, reporting, and follow-up phases. As appropriate, 

surveillance may be employed as part of a TSA. The objective of surveillance is to provide 

confidence through real-time observations that an activity has been performed in accordance 

with approved and specified methods and procedures. It allows for immediate identification of 

any deficiency and initiation of action to correct the deficiency and its underlying cause. When 

non-conformances or deficiencies are identified, project management should be notified 

promptly so that corrective action may be implemented. 
QAPP Handbook 36 December 2012 Agency Review Draft

The QAPP should also state under what conditions surveillance of operations will be performed, 

(i.e., what are the triggering events that would require a surveillance assessment). The 

organization assessed should provide a written response that discusses the action taken to correct 

any observed deficiencies and to prevent similar deficiencies in the future. The QAPP may allow 

for immediate notification of the status and performance of the project to management if 

authorized by the scope of the assessment. It also can include follow-up to verify that corrective 

action was implemented. 

In some cases, activities observed during a surveillance assessment may merit an immediate 

stop- work order. 

The QAPP should discuss the circumstances that would cause a need for such an action and the 

procedure for issuing an official stop-work order (Internal Standard, Clause A3.12 (EPA 2013a).

2.4.4 Audits of Data Quality

An audit of data quality is an examination of data after they have been collected and verified by

project personnel. It is conducted to determine how well the measurement system performed 

with respect to the performance goals specified in the QAPP and whether the data were

accumulated, transferred, reduced, calculated, summarized, and reported correctly. It documents 

and evaluates the methods by which decisions were made during treatment of the data. The 

QAPP should specify the questions to be answered in an audit of data quality such as:

 is there sufficient documentation of all procedures used in the data collection effort to 

allow for repetition of the effort by a person or team with technical qualifications similar 

to those of the original data collector?

 can the data be replicated by the original data collector?

 is there sufficient documentation to verify that the data have been collected and reported 

according to these procedures?

 is enough information provided to allow a potential user to determine the quality and 

limitations of the data and whether the intended use of the data is appropriate? and

 are the data of sufficient quality with respect to DQI goals and other performance criteria 

for their intended use?

An audit of data quality entails tracing data through their processing steps and duplicating 

intermediate calculations. A representative set of the data is traced in detail from raw data and 

instrument readouts through data transcription or transference through data manipulation (either 

manually or electronically by commercial or customized software) through data reduction to 

summary data, data calculations, and final reported data. The focus is on identifying a clear, 

logical connection between the steps. Particular attention is paid to the use of QC data in 

evaluating and reporting the data set. For a large project, a statistical approach may be necessary

to determine a representative number of datasets to be examined. Often, however, the number of 

sets is limited by the budget of the audit. An audit of data quality may occur at different stages of 

the project. For example, it may be performed after field analysis but prior to off-site laboratory

analyses. The products of the data quality assessment are a report detailing the results of custody 

tracing, a study of data transfer and intermediate calculations, a review of QA and QC data, a 
QAPP Handbook 37 December 2012 Agency Review Draft

study of project incidents that resulted in lost data, and a review of study statistics. The audit

report typically ends with conclusions about the quality of the data from the project and their 

fitness for their intended use. The scope and schedule (either by date or triggering event) for any 

audit of data quality should be set in the QAPP.

2.4.5 Qualitative and Quantitative Comparisons to Acceptance Criteria

The data now need to be assessed against the acceptance criteria already established. For each 

DQI or parameter for which acceptance criteria has been established, the data are compared to 

determine how well they meet the criteria. This is done by comparing the individual MQOs to 

the acceptance criteria. 

Qualitative comparisons to acceptance criteria are, almost by definition, very subjective and 

involve the use of expert opinion. In many cases the expert opinion is that of the project manager 

or principal investigator, who may be subjected to unknown bias in evaluating how good the 

comparison is. The principal qualitative indicators are comparability and representativeness. For 

examples on different types of qualitative comparisons, help may be found in Chapter 3, Section 

3.4, which discusses various forms of comparisons when considering integrating existing data 

into a project. Documentation of how the qualitative aspects of comparison of data to acceptance 

criteria have been met is essential.

Appendix B contains supporting information on DQIs for which MQOs can be set as acceptance 

criteria for including existing data in a project. The DQIs that may be able to be quantitatively

assessed are:

 precision;

 bias;

 accuracy (the combination of precision and bias);

 completeness; and

 sensitivity.

A typical example would be to determine if the accuracy of the measuring instrument (precision 

plus bias, the DQI chosen) is at most 10 ppb (the MQO for this DQI). The performance of the 

measuring instrument (actual data) would be used to determine if this MQO could be achieved.

For quantitative comparisons, graphical and statistical analyses can provide estimates of the 

strength of the comparison to criteria. The use of statistical analyses when possible is 

recommended to minimize evaluation bias. The extent of statistical analysis is directly related to 

the wealth of information available about the project; a poorly documented project being less 

useful than a well documented one. For this reason, a well documented QAPP should be 

developed to enable later researchers to evaluate the data for subsequent use. Reference to Data 

Quality Assessment: Statistical Methods for Practitioners, EPA QA/G-9S, (EPA 2006b), a 

guidance document on statistical methods written for non-statisticians, is recommended.

2.4.6 Interim Assessments of Data Quality 

Data quality is formally assessed in the data review steps, which include data verification, data 

validation, and data usability. These steps generally don’t take place until samples are collected 
QAPP Handbook 38 December 2012 Agency Review Draft

and analyzed. However, there are many opportunities during a project to raise possible concerns 

with data quality. Possible indicators of data quality issues include (but are not limited to):

 observed field sampling practices that potentially impact data;

 an anomalous event in the laboratory (e.g., fire or chemical spill) that could potentially

impact data;

 extreme weather (e.g., stream samples collected through a layer of ice); and 

 an intuitive sense that something isn’t right with how the project is proceeding.

The QAPP should include statements encouraging project team personnel to document any

concerns they have over data quality throughout the course of the project. The mechanism for 

making their concerns known, and the individual responsible for determining appropriate action 

should be identified. Interim assessments may also be combined with the verification and 

validation activities for the project (see Section 2.5.1), and with any results from the comparisons 

to acceptance criteria (Section 2.4.5).

2.4.7 Evaluation of Unconventional Measurements

Sometimes there is a need to extend beyond standard analytical methods into unconventional 

methods. These can be thought of as any method for which there is not an already existing 

standard (e.g. EPA, ASTM, or Standard Methods) that has been widely accepted and validated. 

Unconventional methods should not be confused with modifications to an existing method. 

Unconventional methods may include both sampling (e.g. passive sampling for indoor air testing 

using thin films) and testing (e.g. three dimensional gas chromatography) procedures. Typically 

they are developed to sample or measure an analyte that is new or for which a regulatory limit or 

threshold has changed. Many environmental forensic methodologies such as isotope ratio mass

spectrometry, are also considered unconventional because they are not considered to be typical 

for measuring environmental contaminants.

Use of unconventional methods may or may not be allowed depending upon the environmental 

program under which the project is regulated. For some programs only methods that are codified 

are allowed and they must be followed verbatim. If unconventional methods are allowed and 

appropriate for use on the project, the QAPP should include scientifically valid and legally 

defensible documentation of the procedures relating to those methods.

As with conventional methods, the quality of environmental data obtained via unconventional 

methods should support the intended use for the project, and ensure that the DQOs are met. Clear 

MQOs should be established and documented in the QAPP that describe the level of precision, 

bias, and sensitivity expected and needed to meet project objectives. Evaluation of these 

unconventional methods should occur during project assessments, and should be clearly 

documented in the QA Reports and/or final report. It must be remembered, however, that 

research projects often use new methods for which performance data do not exist. A full 

documentation of such methods’ performance may be premature as these methods are amongst 

the first being developed.

Evaluation should include review of at least these items:
QAPP Handbook 39 December 2012 Agency Review Draft

 demonstration that the method can be implemented as described in the QAPP;

 documentation that records of the implementation method are created and maintained;

 how data generated via this method achieves the MQOs set forth in the QAPP; and

 how the data will be used for their intended purpose.

Unconventional measurements come under greater scrutiny than standard measurements and so 

the level of quality assurance should be commensurate with that added scrutiny.

Note that the evaluation of unconventional methods should not occur during assessments by

independent assessors. This activity should have already been performed by project personnel at 

an earlier stage in the development of the project and documented for review by independent 

assessors.

2.4.8 Evaluation of Unconventional Monitoring Projects

Environmental projects span a very wide range of activities and there are some unusual situations 

where the standard methods for monitoring environmental conditions do not fully meet project 

needs. Some examples for which nontraditional environmental monitoring might be required 

include crab counts in the Chesapeake Bay, fish kills in Louisiana rivers, environmental 

sampling for chemical agents surrounding chemical weapons disposal facilities, or monitoring of 

radioactivity levels in pools containing spent nuclear fuel rods. 

As with unconventional laboratory methods, it is especially important that these nontraditional 

monitoring methods be well documented in the QAPP. Along with full descriptions of the 

methods, project-specific MQOs should be stated for representativeness, completeness, and for 

comparability if these data will be combined for use with any other environmental data. For 

some monitoring projects, MQOs for precision, bias, and sensitivity may also be appropriate 

(e.g., the sensitivity of carbon-dating methods for estimating tree age). 

Evaluation of nontraditional monitoring methods should include an assessment implementation 

of the methods against their SOPs or other description in the QAPP. The efficacy of the methods 

should also be evaluated against the MQOs stated in the QAPP. Finally, the usability of the data 

generated from any nontraditional methods should be evaluated to determine if it is of 

appropriate quality to support its intended use on the project.

2.5 REVIEW, EVALUATION OF USABILITY, AND REPORTING 

REQUIREMENTS (ACT)

The final part of the paradigm concerns the actual data; are they valid and can they be used for 

the intended purpose. Verification and validation is followed by an evaluation of usability. 

Included in the usability section is a small discussion of statistical significance. This section 

offers guidance on Standards Clause 7.9, Data Review, Verification and Validation, and Data 

Usability Reporting, Standards Clause 7.10, Documents and Records Management, and Annex 

B3.6, Review, Evaluation of Usability, and Reporting Requirements.

The elements in this section address project checks to see if the data or product obtained will 

conform to the project’s objectives and to estimate the effect of any deviations. These activities 
QAPP Handbook 40 December 2012 Agency Review Draft

may be performed throughout the project to inform mid-course corrections and/or at the 

completion of the project.

Three distinct data evaluation steps are used to ensure that project data quality targets are met

and should be applied to all data collected and used in environmental projects. These steps apply 

to all aspects of data production, including field sampling and analytical activities. Although the 

data verification/validation/usability evaluation process outlined in the following sections is 

portrayed as a sequential process, it may be beneficial (and more cost effective) for many 

projects to combine steps. For example, the entity conducting the verification could also conduct 

the first step of the validation process concurrently. The method by which an organization 

inspects, reviews, verifies, and validates data should be stated in the QAPP.

Data verification entails confirmation by examination and provision of objective evidence that 

the validated information fulfils specified requirements (sampling and analytical), or 

requirements mandated by a contract or agreement.

Data validation means confirmation by examination and provision of objective evidence that the 

particular requirement for which the data or information was collected is fulfilled. It includes the 

process of checking whether the information meets the project’s specifications. 

Data usability is the determination of the adequacy of data, based on the results of verification 

and validation meet the QAPP criteria.

See Appendix D for a more complete discussion of the steps involved in data verification, data 

validation, and data usability.

2.5.1 Data Verification and Validation Targets and Methods

This section describes what information should be included in the QAPP and presents procedures 

for implementing the first two of the three data evaluation steps: verification and validation. 

To perform the data evaluation steps described above, reported analytical data should be 

supported by a complete set of auxiliary (or meta) data as defined in the QAPP. These auxiliary

data include sample receipt and tracking information, chain-of-custody records, tabulated data 

summary forms, and raw analytical data for all field samples, standards, QC samples, and all 

other project-specific documents that are generated. If relevant raw data or sample information 

are not available or adequate to document data quality, then data analysis becomes 

problematical. Lacking a clear linkage of auxiliary data to the project’s objectives creates a 

potential weakness in assessing the quality of the data and the inferences to be drawn from them.

Verification is performed at the onset of the data review process to determine if the required 

information (the complete set of data) is available for further review. It involves a review of 

objective evidence that validated information fulfills specified requirements. The QAPP should 

contain procedures for how verification of field and analytic activities will be conducted.

Validation occurs after verification and ensures that the data and information used in decision

making are of appropriate quality for their intended use. Validation guidance and documents may

be attached or referenced in the QAPP and should address the following:
QAPP Handbook 41 December 2012 Agency Review Draft

 the process that will be used to validate sample collection, handling, field analysis, and 

analytical laboratory project data;

 the specific validation process that will be used for each analytical group, matrix, and 

concentration level;

 the procedures and criteria used to validate data information operations, which may 

include: the electronic or manual transfer, entry, use, and reporting of data for computer 

models, algorithms, and databases; correlation studies; and data plotting;

 how validation of field sampling, handling, and analysis activities will be documented 

such as QC signatures in field logs and QC checklists;

 the person, identified by title, responsible for data validation. 

In some cases the degree of validation is dependent on the organization, program, contract, or 

grantee and consultation with the organization’s QA Manager is advised.

The data validation process is constrained by a number of factors, including contract 

requirements and client/management expectations. However, in most cases there remains an 

opportunity for the data validator or project team to exercise professional judgment in order to 

maximize the benefits of the data validation process. For example, Contract Laboratory 

Program, National Functional Guidelines for Organic Data Review (EPA 1999) includes a 

section titled “Overall Assessment,” describes the use of a brief narrative in which the reviewer 

expresses concerns and comments on the quality of the data. If the reviewer has access to the 

project’s MQOs, then the reviewer is encouraged to comment on the usability of the data. 

2.5.2 Quantitative and Qualitative Evaluations of Usability

Using information provided by the data validator, the project team considers whether data meet 

project quality objectives or DQOs. The usability assessment is the final step of data review and 

can be performed only on data of known quality (i.e., verified and validated data).

Note that this differs from comparisons to acceptance criteria in that here it is the project level 

objectives (DQOs) that are being considered, previously only the specific MQOs were 

considered. A typical example would be to see if the estimated mean level of contaminant (one 

of the project objectives) can be estimated to within 10 ppb (another project level objective) with 

95% certainty (the final project objective). To accomplish this step of data review, the project 

team should do the following and describe the results in the QAPP:

 summarize the usability assessment process and all usability assessment procedures, 

including interim steps, peer reviews, and any statistics, equations, and computer 

algorithms that will be used to assess data;

 describe the documentation that will be generated during usability assessment;

 identify the personnel (by title and organizational affiliation) responsible for performing 

the usability assessment;

 describe how usability assessment results will be presented so that they identify trends,

relationships (correlations), anomalies; and
QAPP Handbook 42 December 2012 Agency Review Draft

 describe the evaluative procedures used to assess overall measurement error associated 

with the project and include the MQOs described in Section 2.2.6.

Statistical comparisons are very useful and are similar to those used with MQOs discussed in 

Section 2.2.6. EPA 2006b may be consulted for examples and instructions of how to perform 

statistical tests. For the success of any project, team members should be encouraged to 

contribute freely in the overall evaluation of the project’s data. 

2.5.3 Potential Limitations on Data Interpretation

The QAPP should address the action to be taken when the MQOs or project-required 

measurement performance criteria are not achieved, if for various reasons, the project data are 

not usable to adequately address environmental questions (i.e., to determine if regulatory or 

project action limits have been exceeded), or the data cannot be used to support project decision 

making.

In addition to potential data usability issues, there are other considerations that might limit the 

use of the data within or beyond the original project. Issues that might impact interpretation of 

data include (but are not limited to):

 unique qualities of the sample matrix;

 inadequate metadata documentation (e.g., temperature, or barometric pressure);

 collection methods that alter the composition of the samples;

 use of analytical methods which may nullify comparisons to standard analyses;

 differences in implied sample support (e.g., different amount of media collected per 

sample, or samples representing different sized areas); 

 extent to which the planned statistical data collection method was compromised using 

judgmental samples; and 

 use of a judgmental sampling scheme (which negates the ability to extrapolate results to a

larger population of interest).

The QAPP should state that any potential limitations of data interpretation should be documented 

in the metadata records associated with these data and included in the final project report.

2.5.4 Reconciliation with Project Requirements

The QAPP should describe how results obtained from the project or task will be reconciled with 

the requirements defined by the data user or decision maker. The proposed methods to analyze 

the data to identify possible anomalies or departures from assumptions established in the 

planning phase of data collection should be included in the QAPP. These should be determined 

prior to data collection to ensure they are driven by project needs. The QAPP should describe 

how reconciliation with user requirements will be documented, issues resolved, and how 

limitations on the use of the data will be reported. Specifically, how sample collection, handling, 

and analysis procedures will be validated against the measurement performance criteria specified 

in Section 2.2.6 should be described. Also, the evaluative procedures to be used in validation to 
QAPP Handbook 43 December 2012 Agency Review Draft

assess overall measurement error associated with the project, including attainment of the 

objectives defined in Section 2.2.6 should be described.

Considering how the data will be used to answer the main study questions is an important step in 

project planning, and should be documented in the QAPP. Without consideration of this, it is 

possible to gather data according to plan and of high quality, but cannot be used to answer the 

questions at hand because no statistical methods are available to support the project needs. The 

QAPP should describe how the data will be summarized or analyzed (e.g., qualitative analysis, 

and descriptive or inferential statistics) to meet project objectives. If descriptive statistics are 

proposed, state, if appropriate, what tables, plots, and/or statistics (e.g., mean, median, standard 

error, or minimum and maximum values) will be used to summarize the data. If an inferential 

method is proposed, indicate whether the method will be a hypothesis test, confidence interval, 

or confidence limit, and describe how the method will be performed. Specifically, the QAPP 

should briefly discuss the five steps of the data quality assessment (DQA) process (see EPA 

2006b) which include:

 a review of the project’s objectives to assure that they are still applicable and a review of 

the sampling design and data collection documentation for consistency with the project 

objectives noting any potential discrepancies;

 a review of preliminary data from QA reports (when possible) for the validation of data, 

calculation of basic statistics, and generation of graphs of the data to probe the structure 

of the data and to identify patterns, relationships, or potential anomalies;

 selection of the appropriate statistical procedures for summarizing and analyzing the data, 

including treatment of censored values (non-detects), based on the review of the 

performance and acceptance criteria associated with the project’s objectives, the 

sampling design, and the preliminary data review, and identification of the key

underlying assumptions associated with the statistical tests;

 verification that those underlying assumptions most likely hold, or whether departures 

could be acceptable, given the actual data and other information about the study; and

 determination of conclusions from the data by performing the calculations pertinent to the 

statistical test, and documentation of the conclusions to be drawn as a result of these

calculations. If the design is to be used again, there should be an evaluation of the 

performance of the sampling design.

Although the steps of the DQA will not be implemented until data collection is complete, the 

final DQA will have comparisons to both qualitative and quantitative criteria. These criteria 

should be documented in the QAPP. Quantitative (statistical) evaluations of the data can provide 

strong evidence in support of, or to refute, project hypotheses or prior assumptions. Even if only

qualitative comparisons are planned, semi-quantitative methods such as graphical methods 

should be planned for in the QAPP.

Statistical significance resulting from quantitative comparisons to planned criteria relies on the 

concept founded on evaluation that the weight of evidence (data) supporting a hypothesis is 

valid. It is never possible to have perfect knowledge about a studied population, but it is possible 

to learn enough about it to be able to say with confidence that a particular hypothesis concerning 

that population cannot be true. However, one should be very careful not to allow the statistics to 
QAPP Handbook 44 December 2012 Agency Review Draft

dictate decisions without recourse to common sense. In particular, as more and more data are 

collected, it becomes easier and easier to achieve statistical significance. The concern is that at 

some point it may be possible to determine statistical significance at levels that are not of 

practical significance. This can be illustrated through the following example:

Based on operations at an industrial plant, and its waste-release permit, it is expected that 

the pH of water leaving the plant will be 5.9. The releases are monitored by weekly 

collections and each week these data are combined with all previous data and the average 

pH is compared to 5.9. After the first few months, the average release pH is 5.88, which 

is not statistically significantly different from 5.9 and the conclusion of no real difference 

justified. After several years have elapsed, the average release pH is 5.8996 and this is 

statistically significantly different from the permitted value of 5.9, but is a conclusion of a 

real difference justified? This is a case where having so much data allows the reviewer to 

identify very small differences from the expected level, but the statistically significant 

result may very well not have any practical significance (in this case a difference in pH of

0.0004, which is barely measurable). 

While statistics provide a strong and essential tool for environmental decision making, the 

science of statistics is not a substitute for common sense and can lead to bad decisions if not 

tempered with practicality. This is particularly true with publications that tend to accept only

papers that show a “statistically significant result” (often using 5% or even 1% as the level of 

significance). It is relatively rare that the publication editor has required the author to include a 

discussion of the practical meaning of the statistical significance using costs, benefits, and 

analysis of the magnitude of the observed effects. The QAPP should take care to specify the 

conditions under which statistical significance may not have practical meaning.

It is worth noting that “statistical significance” should not be the sole indicator of importance of 

a result or conclusion. This was discussed by the U.S. Supreme Court in Matrixx Initiatives, Inc 

v. Siracusano (March 22, 2011) (Ziliak 2011). While the case involved security law, the 

defendant (Matrixx) tried to suggest that its line of demarcation on whether to release data was 

based on statistical significance. The Court disagreed unanimously. They said that the presence 

or absence of statistical significance is not the key factor as to whether an adverse effect is 

material. Justice Sotomayor wrote the opinion noting, “A lack of statistically significant data 

does not mean that medical experts have no reliable basis for inferring a causal link between a 

drug and adverse events”. Statistical significance is only part of the quantitative aspect in the 

interpretation of a project’s results.

Although reconciliation with project requirements represents the final part of the quadripartite 

Plan-Do-Check-Act, it is important to emphasize the final results should be compared with the 

planned objectives of Section 2.2.6. Important deviation of results obtained from those planned 

for can greatly affect the effectiveness of the decisions or inferences made, and can allow the 

project team to make an inherently intuitive conclusion as to the overall validity of results.

2.5.5 Reports to Management

The QAPP should identify the frequency and distribution of reports issued to inform 

management (EPA or otherwise) of project status, including reports on the results of 

performance evaluations and system assessments, results of periodic data quality assessments,
QAPP Handbook 45 December 2012 Agency Review Draft

and significant QA problems, and recommended solutions. Assessment checklists, reports, 

requests for corrective action letters, and the corrective response letters should be included as 

attachments to or referenced in the QA management reports.

Periodic QA management reports are intended to ensure that managers and stakeholders are kept 

informed of project status and results of all QA assessments. Efficient communication of project 

status and problems will allow project managers to implement timely and effective corrective 

actions so that the data produced can meet the project’s objectives.

The following issues may be included in QA management reports:

 summary of project QA/QC program activities including tables, graphs, and charts;

 a summary of training conducted during the project;

 status of project and schedule delays;

 deviations from the approved QAPP and approved amendments to the QAPP;

 description and findings of TSAs and other assessments;

 required corrective actions and implementation; and

 limitations on the use of measurement data generated.

Although these issues listed may be addressed in QA management reports, they should also be 

included in the QA/QC section of the final project report. The final project report should, at a 

minimum, give a reconciliation of project data with project objectives, a data summary

(including tables, charts, and graphs), a summary of major problems encountered and their 

resolution, and any additional data quality concerns together with conclusions and 

recommendations. Separate from reports to management are reports and publications subject to 

peer review, or pre-dissemination review. Formal Agency Peer Review is discussed further in 

Chapter 3, Section 3.4 and also in EPA 2006c. Pre-dissemination review is discussed in PreDissemination Review Guidelines (EPA 2006d).
QAPP Handbook 46 December 2012 Agency Review Draft

CHAPTER 3

QAPP ELEMENTS FOR EVALUATING EXISTING DATA

3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING DATA

In an increasing number of projects, recourse must be made to data other than new data 

produced specifically for this project. The use of existing data (data collected outside of this 

project) can be due to resource or timeliness restrictions, convenience, or may be the only 

potentially suitable data for consideration for this project. Agency Quality Policy CIO 2106.0 

mandates the use of a QAPP: this Chapter gives advice on Standards Clause 7.5, QAPP, and 

Annex B.3 QAPP Elements and Requirements.

Existing data are any data or information available to the project team originally collected for a 

purpose different from the one for which they are intended to be used within the project. They

may be data collected by the same project team previously for another purpose, or they could be 

data from many years in the past that just happen to relate to some aspect of the current project. 

For some programs the data were collected for a purpose similar to the proposed project but by a 

different organization. Existing data (sometimes erroneously known as “secondary data”, and 

occasionally discussed as “secondary use of data”) are challenging to use because determining 

whether they are of appropriate quality can be difficult to ascertain. 

Existing data are often used because they may be easily accessible, less resource-intensive than 

collecting new data, available within a short time-frame, or even collected for a similar project to 

the one the dataset is being considered for. Some examples include data obtained from 

www.data.gov; published literature, reports, and handbooks; state and local programs; outputs 

from existing models; pilot studies; GIS layers; surveys; and other Government agencies.

Inspection of existing data for potential use can be greatly helped by examining the metadata that 

accompanies the dataset. Metadata is the information that describes the dataset and its quality

criteria. The ultimate success of an environmental program or project depends on the quality of 

the environmental data used in decision making, and this quality depends significantly on the 

ability to inspect the metadata and ascertain its applicability to the project. It will be rare that the 

existing data set will be exactly what is needed for the proposed project, and documentation of 

the difference between project requirements and existing data characteristics is essential. It is 

important to utilize the graded approach which allows the existing data set to be considered in its 

entirety rather than by individual characteristic unless the objectives of the project demand it. 

Probably the most important part of the QAPP is in the planning for obtaining the data or 

information because an error at this stage may be difficult to rectify once the project is 

underway. One of the key outputs of the systematic planning process is the construction of a 

conceptual model that describes the scientific and engineering process under investigation. The 

conceptual model is an important tool for organizing information about the current state of 

knowledge and understanding of the project, as well as for documenting key theoretical and 

practical assumptions underlying the data and information collection. Further details on 

systematic planning are to be found in Annex C of the Standards and EPA 2006a.
QAPP Handbook 47 December 2012 Agency Review Draft

QAPPs must, at a minimum, address all elements required by the Standards and these elements 

are described later in more detail with guidance for how to implement them. The QAPP is 

invaluable in not only documenting all aspects of the project but serves as a structured resource 

for writing the final report on the project.

The elements arranged according to the Plan-Do-Check-Act project life cycle, are described in 

the following sections. 

3.2 PROJECT MANAGEMENT (PLAN)

Planning, the first part of the construction of a QAPP, deals mainly with the management 

aspects of the project. Most of the structure is simple good management practices, the only QArelated part being the establishment of data or project quality objectives and measurement 

performance criteria (section 3.2.6). The section offers guidance on Standards Clause 7.5, 

QAPP, Standards Clause 7.7, Systematic Planning, Standards Clause 7.10, Documents and 

Records Management, and Annex B3.3 Project Management, and Annex B3.4 Data Acquisition. 

The elements in this section address the format and disposition of the QAPP, project 

administrative functions, project information, and goals. These elements document the backbone 

of the project-planning process and lay the groundwork for the more technical elements.

The QAPP must describe the project adequately and the elements that can address the basic 

project management and objectives of the work include:

 title, version number, and approval/sign-off sheets;

 document format and table of contents;

 distribution list;

 project organization and schedule;

 project/problem background and description;

 data or project quality objectives and measurement performance criteria; 

 special training requirements/certification; and

 documentation and records requirements.

3.2.1 Title, Version, and Approval/Sign-Off 

Each QAPP should include a page with the title of the project and the name of the organizations 

involved in various aspects of that project. The version of the QAPP should also be clearly 

identified along with the title. It is acceptable to create separate title pages and signature pages, 

as long as the document title, version number, and date appear on the signature page. The names, 

titles, signatures, and signature dates of those approving the plan are also placed on this page. 

Individuals responsible for approving the QAPP may include the organization’s Technical 

Project Manager and QA Manager, and the EPA (or other funding agency) Project Manager and 

QA Manager. Their signatures indicate both their approval of the plan and commitment to follow 

the procedures noted. Other key personnel that may sign the plan are other QA officers, prime 

contractors, and subcontractors.
QAPP Handbook 48 December 2012 Agency Review Draft

This approval information is typically the first page of the QAPP. Depending on the 

organization’s administrative policy, QAPP approval could also be in a separate memorandum. 

The signature dates indicate the earliest date when the environmental data operations portion of 

the project can start (i.e., its effective date). The QA Manager will also determine if digital 

(electronic) signatures are acceptable for the approval of the QAPP.

In addition to the title, version number, and approval signatures, it is important to include a 

revision history. Each time the QAPP is revised, as approved by the QA Manager, the version 

number should be updated and the revision history should be amended to include a brief 

summary of the change and date. 

3.2.2 Document Format and Table of Contents

The QAPP should be organized such that it meets the project’s needs, can be reviewed 

efficiently, and meets the document control requirements of the QMS under which it is 

developed. A document control format, such as the example shown in Figure 5, may be used to 

support QAPP development, or a footer created for each page to show revision status. 

Figure 5. Document control format example

The Table of Contents will generally list QAPP elements, as well as any tables, figures, reference

section, and appendices necessary to the project. If the QAPP author prefers to organize the plan 

differently than how the elements are organized in this Handbook a table may be inserted here to 

cross-reference where the information for each element may be found to simplify review. 

Depending on the type of project, analytical research protocols, or data management procedures 

may be attached. If the existing data are unaccompanied by SOPs, resort to other indicators of 

quality may have to be made and documented. In the case where proprietary standards (for 

example, those of ASTM or ISO) are used, reference to the location on the relevant website may

be sufficient.

3.2.3 Distribution List

The distribution list identifies all individuals who should get a copy of the approved QAPP, 

either in hard copy or electronic format, as well as any subsequent revisions. Key personnel 

responsible for project implementation and funding, and who should have the currently approved 

QAPP, should be listed with their project titles or positions, organization names, email addresses, 

and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will 

need access to it, the distribution list also serves as an easy reference of who needs to be alerted 

and provided with a revised version of the QAPP in the case that modifications are necessary. 

Some organizations choose to provide the distribution list on the title or approval page, others 

elect to include this list in the project organization section when listing key personnel and their 

contact information.

Project Name/# __________________

Section # _______________________

Revision # ______________________

Date ___________________________

Page ___________ of ____________
QAPP Handbook 49 December 2012 Agency Review Draft

3.2.4 Project Organization and Schedule

It is important that roles and responsibilities are well defined prior to initiating project activities. 

Those individuals involved with the major aspects of the project are named in this element along 

with their project responsibilities. For example, the people responsible for maintaining, updating, 

and overseeing implementation of the QAPP would be named here. The personnel included in 

this element should include the lead scientists, researchers, modelers, consultants, and contact 

information for the external organization holding the existing data. If actual personnel cannot be 

initially identified, then the position description of that person’s function should be given. 

An organizational chart or table can be very helpful, and should be included if appropriate. It is 

also helpful to indicate lines of communication among individuals or groups, and this can be 

shown easily on an organizational chart or an organizational network diagram. While a single 

individual may have more than one responsibility in a project, the project should be organized 

such that any person having QA responsibilities is independent of those generating and using 

project information. If this is not possible, an alternative method of ensuring effective QA review 

should be specified in the QAPP.

The level of detail included in the schedule is left to the discretion of the QAPP authors. It may

be beneficial to have a very detailed and strongly stated schedule for the project to follow. In this 

case, there is a risk of requiring QAPP revisions if the schedule needs to be changed during the 

lifetime of the project unless the QAPP specifically states it will not be revised simply due to 

schedule delays. It is useful to include critical points in the project such as expected date of 

QAPP approval, sub-section start and end points, or dates modeling subroutines need to be 

completed. When creating the schedule, allowance should be made for potential delays and 

general inefficiencies inherent in any project. If the project includes regulatory or courtmandated deadlines, these should be highlighted to ensure their importance is noted. For projects 

in which deadlines or milestones are not well defined, a more generalized work schedule can be 

formulated.

3.2.5 Project Background, Overview, and Intended Use of Data

This overview should give the reader an understanding of the problem to be solved, along with 

any pertinent background information for the project. It describes why the project will be done

and what goals the project intends to accomplish. The general project goals stated here form the 

foundation for the entire study. Equally important, the development and documentation of this 

element will help ensure that all project team members clearly understand and agree on the 

underlying purpose of the project, increasing the likelihood that the project design will address 

and accomplish that purpose.

In addition to the general overview, include information on the background and use of previously

approved QAPPs relevant to this project. In addition to information about the project, it is good 

practice to include an outline of information that is currently not sufficiently developed. Clearly 

state who needs the information and what the intended use of this information will be. Problems 

that are more complex will lead to more extensive information in this section. The reader should 

be able to understand the importance or context of the project. The general project goals stated in 

this section will be refined in Section 3.2.6.
QAPP Handbook 50 December 2012 Agency Review Draft

3.2.6 Data/Project Quality Objectives and Measurement Performance Criteria

In this element, the QAPP should indicate how good the data or information must be to answer 

the question, resolve the problem, or support the decision to the level desired for this project. For 

projects that use existing environmental data, acceptance or performance criteria for the data 

should be determined during the project planning process and documented in the QAPP.

EPA requires the use of systematic planning (see also Annex C of the two Standards) for all 

projects that involve the collection or use of environmental data and the QAPP documents the 

outcomes of this process. When using existing data it is very important that a systematic 

planning process has been used as it directly influences the quality and representativeness of the 

data generated. EPA encourages the use of the DQO process, which may also be called project 

quality objectives, depending on the organization’s preference. The DQOs are established from 

total study variability as the overall qualitative and quantitative goals of the project. 

Under any systematic planning process for a project that includes environmental data collection 

it is desirable to define tolerable levels of uncertainty for components of the total study

variability. Total study variability is due to natural (field) variability and measurement 

(laboratory) variability. The effects of field variability depend on the selecting of an appropriate 

sampling design and number of samples collected (see Appendix B for further discussion). 

Measurement variability depends on the specific measurement and analytical techniques (see 

Appendix C for further discussion). 

DQOs are supported by Data Quality Indicators (DQIs), which are supported by Measurement 

Quality Objectives (MQOs). These DQIs are linked to field and measurement variability (see 

Appendix B for further discussion). MQOs are sometimes known as Measurement Performance 

Criteria (MPCs) depending on the protocols of the organization. See Appendix B for definitions 

and details of DQIs and MQOs. It is essential that the definition of what is meant by MQOs or 

DQOs is made clear early in this section. Some organizations use the same acronyms for 

different operations. For example, some organizations use DQOs in place of DQI performance 

measures, or use MPCs for these measures.

The traditional DQIs considered in project planning include:

 precision;

 bias;

 representativeness;

 comparability;

 completeness; and

 sensitivity.

While historically there has been considerable attention directed to bias, precision, and 

sensitivity, it is really representativeness that is probably the single most important indicator of 

data quality. Representativeness is a qualitative measure of the degree to which data accurately 

and precisely represent a characteristic of a population. 
QAPP Handbook 51 December 2012 Agency Review Draft

A poor sampling design with very high quality analytical laboratory analyses might give data 

that meet MQOs for precision, bias, sensitivity, comparability, and completeness while not 

providing information that is actually representative of the characteristics of the population of 

interest. If the existing data fail to be representative of what was intended to be sampled, no 

amount of statistical manipulation can make the data set valid. Great attention should be given to 

ensure that the existing data are truly representative of the project’s population of interest. It is 

axiomatic that a sufficient number of samples will be needed to assure a good representation of 

the population has been achieved.

The QAPP should indicate how rigorous the existing data or information has to be to answer the 

question, resolve the problem, or support the decision to the level desired and needed for the 

project. Information on acceptance or performance criteria should be determined during the 

project planning process and documented in the QAPP. 

Establishing DQOs and MQOs for exploratory projects (for example, research studies) where 

little or no information on the quantitative magnitude of what is expected is difficult. The use of 

information from similar or related studies is useful. A discussion documented in the QAPP of 

the consequences (or results) of using existing data with unknown MQOs is advisable.

3.2.7 Special Training Requirements and Certification

It is not likely that special training will be required when using existing data, but it is still 

possible (e.g., handling of Confidential Business Information, or security clearance). The QAPP 

should also specify how this information will be documented, where the records will be kept, and 

indicate who is responsible for ensuring that these special training and certificate needs are met 

and documented. 

In some cases, reference to an outside authority or consultant with the skills and training 

appropriate for the evaluation of a specific scientific, engineering, modeling, or other technical 

area may be necessary. The name of the consultant together with a summary of their field of 

expertise should be documented in the QAPP.

3.2.8 Documentation and Records Requirements

The QAPP should describe the process and responsibilities for ensuring that project personnel 

will receive the most recently approved project documents such as the QAPP, SOPs, and other 

documents used throughout the project operation. QAPP authors should discuss how these 

documents will be updated and how information regarding updates will be communicated. 

The QAPP should indicate where all project documents such as the QAPP or final report and 

records such as statutory requests and training records will be stored and for how long. In some 

cases retention, access to, and final disposition of some records may be regulated. In those cases, 

this element should address and comply with all relevant regulations. 
QAPP Handbook 52 December 2012 Agency Review Draft

3.3 DATA ACQUISITION (DO)

This is the heart of a QAPP as it documents the requirements for dataset identification and 

access, the establishment of acceptance criteria, information on how the data were collected, 

and how the integrity of the data are assured. This section offers advice on the Standards Clause 

2.1, Scope, Clause 7.7, Systematic Planning, and Annex B3.4, Data Acquisition. 

3.3.1 Proposed Data Source Originator and Publication Information 

Knowing who originated (initially collected, who generated, or was responsible for) the existing 

data can be very informative. Data originators might be EPA, states, tribes, municipalities, 

potentially responsible parties (PRPs), or activist groups, for example. Confidence in the data 

may, to some small extent, be judged according to the data originator. That is not to say that data 

collected by states, for example, should always be accepted, or that data collected by PRPs, for 

example, should always be excluded. However, knowing who the data originator was may offer 

some information as to the purpose of the data collection and reflect on the possible impartiality

of the lead author or source of the data.

Whether data are private or publicly accessible, and if they have ever been published, may also 

be helpful to know. Conversely, if the data are identified without any understanding of who 

originally collected them or for what purpose, that would be cause for very careful consideration 

of their appropriateness for use in the project. 

The QAPP should discuss how the data source originators will be identified and documented, 

whether or not the data have previously been made public, and how any decisions regarding the 

use of data will be dependent on this information. The rationale for the selection of a particular 

database in preference to others should be discussed with respect to the acceptance criteria 

established for this project (see Section 3.3.3). If one dataset is considered “higher quality” than 

another, then the rationale for this decision must be documented in the QAPP. 

In some cases, the site where the dataset can be found is of importance when evaluating the 

suitability of the data for use. As an example, data available from a Government site would have 

been subjected to some set of quality assurance protocols (see section 3.4.1), whereas a site that 

does not practice or document their quality assurance protocols would be of lesser possible 

worth.

3.3.2 Data Format and Accessibility

The QAPP should describe, to the extent possible, how the existing data will be accessed and 

what format they are in. Data are stored in various ways ranging from national relational 

databases where access is relatively easy, to stacks of documents in the back corner of a dusty 

warehouse. Any knowledge regarding data accessibility and format should be described in the 

QAPP, and if that is unknown, how it will be ascertained may be included. Procedures for 

transferring the data to a format usable by the project team should be discussed if relevant. For a 

further discussion see Brilis et al. 2004.
QAPP Handbook 53 December 2012 Agency Review Draft

3.3.3 Establishment of Acceptance Criteria

Section 3.2.6 discussed the need to document DQOs that apply to the project as a whole. 

Acceptance criteria are the specific quality needs for existing data. In many cases these may be 

universal (e.g., the same requirements may be applicable for all existing data brought into the 

project). There may also be cases where the acceptance criteria for a particular set of existing 

data have slightly different acceptance criteria than for other data brought into the project (e.g., 

the compiled data in full should be representative of the whole watershed, but this particular data 

set is meant to be representative of only the area at the mouth of one specific creek). The QAPP 

should document the acceptance criteria for all existing data that will be included in the project.

The use of assumptions to qualify the existing data set as adequate for use should be carefully

documented as these can have an immense impact on the validity of the final project’s results if 

unknown bias is present. The use of assumptions such as “presumed to be valid”, or “assume this 

data can be used for this project” is severely discouraged and effort should be made to establish

objective acceptance criteria.

It must be acknowledged that when considering existing data for potential use, quantitative 

acceptance criteria may be difficult to construct. However, some concept of what the acceptable 

range of acceptance criteria needs to be developed by the project team. Reliance on broad 

statements such as “we’ll use the best we can find” or “whatever is in the literature” should be 

discouraged. If the quantitative criteria are poorly defined, then an iterative approach may be 

appropriate where the characteristics of one existing data set are compared to those of another 

until the “best” is selected. It is advised that the acceptance criteria be phrased in a comparative 

fashion by selecting an important DQI and then addressing the MQO for that indicator. For 

example, “the dataset having the most precise estimate of variance will be selected”, the desired 

indicator, precision, is the acceptance criterion for the different datasets under consideration.

When possible, the existing data should be compared to the MQOs (or MPCs depending on the 

nomenclature of the project’s organization) related to the objective of the project through 

examination of the metadata that accompanies each project. The principal DQIs (precision, bias, 

representativeness, comparability, completeness, and sensitivity) are especially useful in 

establishing an overall set of descriptors of evaluators of existing datasets. Other contributory

indicators such as age of data, temporal representativeness, technological obsolesce should be 

developed if relevant and documented here. When using data from a variety of sources or 

sampling events, it is important to be sure that the data are in harmony (e.g., similar sample 

collection design, sample collection methodology, sample preparation, analysis, and reporting) in 

the ways useful to the project’s objectives. When an existing dataset lacks information on 

MQOs, careful attention should be given to the overall DQOs of this dataset, because the set of 

MQOs directly affect DQOs. A very “tight” achievement of the DQOs would be a good augury

of the quality of the dataset, a “looser” achievement, less so. In general, the QAPP should foster 

a “common sense” approach in the absence of the possibility of setting quantitative objectives 

and consequent reliance on qualitative objectives. When metadata are absent, recourse to 

statistical analysis or comparisons to similar studies may be necessary and then documented.

In some circumstances, DQOs may be stated in qualitative terms having definitive points of 

reference.
QAPP Handbook 54 December 2012 Agency Review Draft

For example, “The existing data qualifies for the ‘A’ rating with respect to the rating system 

described in Section 4.4.2 of the Procedures for Preparing Emission Factor Documents (EPA454/R-95-015)”. Although it is stated in qualitative terms, this DQO is measurable using specific 

acceptance criteria that are presented in the cited document. Simplistic descriptors such as “very

good” or “acceptable for this project” are to be discouraged unless these are specifically defined 

in the QAPP.

Qualitative acceptance criteria (for example, representativeness, or comparability) may be 

difficult to describe, but need to be couched in terms that clearly indicate why one set of existing 

data may be preferred to another. For example, “the project’s objectives require the cohort of 

data be exposed to defined environmental conditions for a period not exceeding 5 years”, the 

selected indicator, comparability, being linked to the objectives of the project.

3.3.4 Sample Data Collection Methodology

The QAPP should give great attention as to how the existing data set has been collected with 

particular reference to whether a statistical (probabilistic) or judgmental data collection scheme 

was used. The inference or conclusions from the data hinge on this distinction.

There can be a tremendous problem if judgmentally sampled data are misclassified as random (of 

some description) data as the probabilistic statements generated are unlikely to be valid. The 

same is true if data consisting of composite samples are regarded the same as if they were 

random samples. The problem is compounded when data from different collection or analysis 

are compared or evaluated against acceptance criteria. 

For example, are all data from the top six inches of soil, or are some of the data from deeper 

intervals? Were all data collected according to statistically rigorous sampling designs such that 

inferences to the greater population can be made, or were some of the data collected based on 

best professional judgment? Do all data represent the same amount of environmental media, or 

are some samples simple grabs while others are composite samples? These and many other 

similar questions shed light on the appropriateness of combining data and using them for a 

common purpose. The QAPP should identify and document what is important to the project team 

relating to sample data collection methodology.

3.3.5 Quality Program and Quality Assurance Procedures Used by Data Originator

If possible, the Quality Management Plan (QMP) under which the existing data were collected, 

and the QA procedures that were used during data collection should be documented in the

QAPP. At a minimum, the quality program should be referenced, and some documentation such 

as a QMP or text extracted from the QMP may be needed as an attachment to the QAPP to 

provide evidence that the data quality are expected to meet the project’s objectives. 

When the quality program under which the data were collected is unknown, discussion of how 

the project will deal with unknown data quality should be documented to avoid unacceptable 

uncertainties in project decisions that are based on existing data. If the quality program under 

which the data were collected is unknown when the QAPP is written, but is later identified, it 

should either be included in a revision to the QAPP or documented in the final report.
QAPP Handbook 55 December 2012 Agency Review Draft

It is, however, rare that existing data have the QC information readily at hand although it is quite 

possible the QC procedures would be documented. The project team should carefully consider 

the importance of each type of information and determine how its absence will impact the project 

if this information is not available. Data from EPA should have this information as part of the 

original QAPP as QC procedures are an integral part of the QAPP, but is less likely for other 

organizations. Effort should be made to ascertain the status of the data generator’s QMP as it is 

possible that QC procedures would have been referenced as part of the analytical SOW, thus 

giving more credence to the reliability of the data.

Regardless of whether the quality program under which the data were collected is known, this 

section should include documentation of the project’s data quality requirements along with 

statements of how the data quality will be assessed.

The overriding objective in considering the Quality Program under which the existing data was 

collected is ascertaining the integrity of the data. It is important to ask the question “Can I trust 

these data?” together with the follow-up question “What leads me to believe these data are 

reliable?” Documentation of qualitative (and quantitative if possible) aspects of the dataset that 

answer these questions is essential in ensuring the dependability of these data.

3.3.6 Documentation of Sample Quality Assurance Procedures

Section 3.3.5 discussed documentation of the QA procedures used when the existing data were 

originally collected. This section is for QA procedures to be used as existing data are brought 

into the project. For example, if the existing data are available only in pdf files and need to be 

converted to spreadsheets for analyses, the procedures used to assure that errors in translation are 

minimized should be documented here. Some datasets may be subject to regulations such as the 

special rules governing certain information (40 CFR 3) and may have the needed information. In 

general, the QAPP should specify what procedures will be used to ensure that existing data 

maintain their original integrity and quality as they are migrated into current project files.

For existing data to be used in the project, the QAPP should ideally identify the information to be 

included in the project data files (project record) and its format, if possible.

3.3.7 Data Management Requirements

How the existing data will be stored, who will be responsible for access and maintenance, and 

how it will be incorporated with other project data are important elements to consider. 

The element is meant to provide an opportunity to document the hardware, software, and 

personnel requirements for managing and incorporating existing data into the project. All aspects 

of data management, including handling, tracking, processing, and data transfer need QC-type 

measures to ensure the integrity and dependability of the data is not lost. This would apply

equally to both manual and electronic information systems and transfers between them.

The security of the database is one QC-type measure that helps maintain the integrity of the data. 

Access to the database with the ability to edit the data is a potential threat to integrity and the 

measures taken to prevent unauthorized or accidental editing of data should be documented.

Project teams are also encouraged to retain information about the original sources of the data in 

the actual database so this information is available for future analysis.
QAPP Handbook 56 December 2012 Agency Review Draft

3.4 ASSESSMENTS (CHECK)

The third part of the Plan-Do-Check-Act paradigm considers the qualitative and quantitative 

aspects of QA that apply to ensuring the integrity of the existing data. The techniques used for 

generating new data rarely apply as existing data rarely are accompanied by detailed QC 

information. When making assessment on existing data attention has to be given to careful use of 

the graded approach that selects the level of assessment commensurate with the intended use of 

the data. In some cases a “judgment call” will have to be made as the existing data may not have 

the detail desired to enable a good assessment to be made; documentation of the rationale for 

making the decision to use the dataset is essential. The advice given here is drawn from the 

Standards Clause 7.8, Assessment of Data and Information, and Annex B3.5 Assessments.

For existing data, technical systems audits, performance audits of measurement and analytical 

systems, surveillance of operations, and audits of data quality usually cannot be conducted unless 

QC information is available.

3.4.1 Qualitative Comparisons to Acceptance Criteria

Qualitative comparisons to acceptance criteria are, almost by definition, very subjective and 

involve the use of expert opinion. In many cases the expert opinion is that of the project manager 

or principal investigator, whose judgment may be biased. Documentation of how the qualitative 

aspects of comparison of data to acceptance criteria have been met is essential. The two principal 

DQIs, comparability and representativeness, are the most important indicators to consider. The 

following table shows examples of various aspects of qualitative comparisons: 

Table 1. Examples of Qualitative Comparisons

Example Indicator of 

Comparability

Example Questions and Answers for Comparison to Qualitative Criteria

Samples within 

datasets should be 

selected in a similar 

manner 

Are the sample collection designs or methods compatible? If a 

probabilistic sample design has been used, ensure they are the same 

type (e.g. random). If stratification has been used make sure the 

boundaries of the strata (sub-populations) are comparable. If a complex 

design has been used, assistance of a statistician may be necessary.

Were the sampling locations or frequencies selected in a similar 

manner? If non-probabilistic (directed, or non-random) samples were 

taken then there needs to be a justification of why these samples are 

acceptable for use in the project. Caveats may be necessary as the 

conclusions from non-probabilistic are difficult to extrapolate to wider 

situations of even populations of interest. If the non-random samples are 

arguably similar to what would have been selected if a probabilistic 

sample had been used then a justification should be made. If a 

probabilistic sample has been augmented by non-random samples, a 

minor justification should be made if the augmentation is small, but a 

more comprehensive justification would be needed if the augmentation 

is relatively large. 
QAPP Handbook 57 December 2012 Agency Review Draft

Example Indicator of 

Comparability

Example Questions and Answers for Comparison to Qualitative Criteria

The samples should 

apply to the same 

target population

Are they equally representative of the population of interest? Attention 

must be given to ensure the two sets of samples really apply to the same 

target population; consideration of the metadata contained in the 

description of how the samples were obtained is essential. It is advisable 

to evaluate any differences with respect to potential bias in the 

conclusions drawn from comparisons of the two datasets or conclusions 

from combining the two datasets.

Data should be 

temporally and 

spatially consistent

Were samples collected in the same sampling event? If there are 

discrepancies in time, justification of why these time periods are of little 

importance will be needed. In some cases it may be necessary to argue 

these time periods are essentially “typical” of the situation.

Are there temporal factors such as seasonality? This may be of 

importance and a justification of why this can be ignored will be 

needed. For example, comparing stream flow data from the winter 

season with that of the summer season may not make sense due to the 

different seasonal factors influencing stream flow.

Are the samples taken representative of the same population? This may 

require a discussion on what representativeness really means in context 

with the project’s goals and objectives. There is no easy definition of 

representativeness and so justification using qualitative measures will 

be necessary.

Were the samples in comparable matrices? The efficacy of extraction of 

contaminates from the sample obtained from the field is very dependent 

on how the contaminant binds or adsorbs to the inert materials of the 

matrix. Comparison of data obtained from a water sample with that 

from sediment may be problematical depending on the ease by which the 

contaminates can be measured. Reference to methods handbooks may be 

necessary to ascertain comparability.

Datasets should 

contain the same set 

of variables of 

interest

Were the variables of interest reported for all datasets? Sometimes 

measurements on key variable are not available. If only a few readings 

are missing it may be possible to justify imputation methods to substitute 

a value using information from auxiliary variables. This is particularly 

true when comparing or merging multivariate datasets. It depends on 

what information is available from the metadata associated with the 

data set. For example, particle size, total organic carbon, or percent 

moisture may be useful for determining if the data are comparable, but 

these variables may not always be reported.
QAPP Handbook 58 December 2012 Agency Review Draft

Example Indicator of 

Comparability

Example Questions and Answers for Comparison to Qualitative Criteria

Units in which these 

variables were 

measured should 

convert to a common 

metric

The units of measurement should be reported for all datasets. This goes 

beyond a simple Imperial/Metric (ounces to grams) argument but to 

whether the units all convert to a common metric. For example, some 

results may be reported in wet weight and some in dry weight, which are 

not directly comparable without additional information. How this 

information is used becomes part of the justification.

Field collection 

methods should be 

similar

What instrument was used and which procedure was followed? It is 

important to note the way in which the physical samples were collected 

as different methods may lead to different results. For example, 

documenting the actual collection method from a 6” core could be very 

different from that obtained from a composite sample of four 6’ cores 

that have been combined to make a single sample. Incremental samples 

(often between 50 and 200 very small samples combined into a single 

sample) should be examined with respect to the project objectives. The 

QA/QC criteria used in making the incremental sample must be 

documented to ensure the use of this information matches the goal of the 

project. Inadvertent bias may result from the merging of data from 

differing sample collection methods and justification for use may have to 

be comprehensive. The same is true of comparing filtered samples with

unfiltered samples, or where the sample handling conditions are 

markedly dissimilar. 

Rules for excluding 

certain types of 

observations should 

be similar for all 

datasets

Are “outliers” treated in a similar fashion across all datasets? The 

exclusion of data for being ‘too large’ or ‘too small’ is a potentially 

biasing situation. The reason for the exclusion of data must be carefully 

examined by the potential user as incorrect exclusion can create some 

extreme bias. The rationale for exclusion must be carefully considered 

with respect to the goals and objectives of the project. If no documented 

rationale exists, then a justification for its use is essential.

The nomenclature 

should be consistent 

with the objectives of 

the project

Are the definitions used in the existing dataset consistent with those of 

the project? Convertible units (e.g. Imperial to Metric or g/Kg to ppb)

are not the problem but definitions involving the target population and 

sample technique used could cause problems if too divergent. 

Metadata should be 

adequate to 

understand the 

protocols used

Are the metadata (information about the way the actual data were 

obtained) adequately described? The metadata should describe how the 

samples (data) were collected and be comprehensive enough for a 

reviewer to understand the purposes and procedures involved. 
QAPP Handbook 59 December 2012 Agency Review Draft

Example Indicator of 

Comparability

Example Questions and Answers for Comparison to Qualitative Criteria

Similar analytical 

methods or 

procedures should be 

used to analyze the 

samples for all 

datasets. 

Was the same analytical method used for all sample results considered 

and, if not, are the analytical methods comparable? The use of routine 

methods simplifies the determination of comparability because all 

laboratories used the same standardized procedures and reporting 

parameters. However, when reviewing the analytical methods used to 

produce the data, consideration should also be given to options in 

methods that may have been used by the laboratory. Although the 

analytical method may be the same, options such as matrix or 

concentration level will affect results reported. For difficult matrices, 

different methods are sometimes used to obtain measurements of 

different analytes. If different methods have been used, a justification as 

to why the data are suitable for the project may be necessary. The same 

is true for the preparation methods used before the actual sample was 

analyzed. For example, data resulting from a total digestion preparation 

when compared with data from a partial digestion preparation would 

not result in directly comparable results. 

Measuring devices 

used for both datasets

should have 

approximately similar 

detection levels

Are the detection or quantitation levels acceptable for use in the 

proposed project? Combining datasets having different detection or 

quantitation levels can lead to difficulties in analytical interpretations. 

For example, data set A may have a minimum detection of 5.0ppm, but 

data set B a minimum detection of 2.0ppm. Justification will be 

necessary to describe how data occurring between 2.0 and 5.0 in data 

set B will be treated when both datasets are merged together. 

Qualitative evaluations of data having been through a peer review system are important when 

only partial information is available about a data set. In most cases existing data have been 

through some form of review and it follows that publication resulting from an incomplete 

evaluation of data used should be accompanied by a disclaimer. Given that existing data have

cleared some pre-requisite before release, there are four cases to consider; data occurring in a 

peer-reviewed publication, data from other Government organizations, data from within the 

EPA, and data that have been accepted by the courts.

Peer Review for Publication

In the case of papers, articles, and discussion pieces written for external publication, the peer 

review procedure used is determined by the individual publication’s review policy. This can 

range from a simple acceptance by an editor for conformance to accepted practices for that 

publication, to double-blind review with full documentation for extremely rigorous publications.

It is usual for most publications to employ an anonymous reviewer (referee) system (the reviewer 

does know the identity of the paper’s author, hence single-blind review) with at least one level of 

review by peers competent in the same field as the author.
QAPP Handbook 60 December 2012 Agency Review Draft

In many scientific publications the authors make some statement on the QC procedures used to 

generate the data but rarely give specifics. Additionally, they only occasionally make available 

the original data to the peer reviewers. A summary of the data or the data summary statistics 

usually appears in the published article. In many cases this is not due to the author’s wish to 

prevent the release of data (although this could be true in the case of proprietary data or data 

deemed to be of a sensitive nature), but an economic one based on publication printing and 

distribution costs. For a publication constrained by internal policy not to exceed a certain 

size/distribution, this could be the over-riding factor in causing lack of data availability. Some 

publications make the data accessible to other interested parties for testing for internal 

consistency and reliability, but that it is not a universal practice.

In some instances a request to the principal author can result in the release of data in sufficient 

detail that comparison to a set of acceptance criteria may be made. If however, the data fall under 

proprietary information such as Confidential Business Information, even a request to the 

principal author may not result in the release of data in a useful format.

Consideration and evaluation of whether the data meet the Acceptance Criteria established by the 

potential user of the data must still be determined and documented. The caveat that a full 

investigation of data characteristics could not be done should accompany the evaluation.

Data from other Federal Organizations

When considering the use of data from another Federal agency or organization, there is a 

minimum set of peer review requirements established. Nearly all Federal organizations have to 

comply with specific peer review requirements before disseminating influential scientific 

information and data and make available relevant data. These requirements were published by 

the Office of Management and Budget (OMB) in a Peer Review Bulletin issued December, 2004

(OMB 2004), and provides the definition of influential scientific information as:

“Scientific information the agency reasonably can determine will have or 

does have a clear and substantial impact on important public policies or 

private sector decisions. ...(T)he term “influential” should be interpreted 

consistently with OMB’s government-wide information quality guidelines 

and the information quality guidelines of the agency.”

The Bulletin provides detailed peer review guidelines and applies a more stringent set of 

requirements for “highly influential scientific assessments”, which are further defined as:

“...an evaluation of a body of scientific or technical knowledge that 

typically synthesizes multiple factual inputs, data, models, assumptions, 

and/or applies best professional judgment to bridge uncertainties in the 

available information.”

The difference from a traditional peer review for publication is that there is public disclosure of 

peer reviewers’ identities when dealing with highly influential scientific assessments. In addition 

it is required that the peer reviewers, their reports, and the response of the agency to those reports 

be made available to the public. 

The evaluation of whether the data meet the acceptance criteria established by the potential user 

of the data is made easier when peer review reports can be considered with the published data 
QAPP Handbook 61 December 2012 Agency Review Draft

and information. The decision of whether the data meet the definition of “influential” (thus 

triggering the peer review requirements) has to be documented and use of data not defined as 

influential justified. 

Data from EPA

The peer review requirements for Agency data are laid out in the Peer Review Handbook (EPA 

2006c). It follows the guidelines established by the OMB Bulletin and requires that influential 

scientific information be peer-reviewed in accordance with the handbook. The handbook 

provides a flowchart for conducting a peer review and includes a checklist for managers planning 

a peer review. The handbook notes “Peer review enhances the credibility and acceptance of the 

decision based on the work product” and encourages the widespread use of peer review. 

According to Stephen L. Johnson, EPA Administrator in 2006, “Peer review of all scientific and 

technical information that is intended to inform or support Agency decisions is encouraged and 

expected”(EPA 2006c).

Information and data generated by the Agency should be easier to evaluate against the 

acceptance criteria established by the potential user of the data. Informal collaboration or 

assistance from the data generator is often available, leading to a more comprehensive evaluation 

for potential use. A request for information in the form of the QAPP used to collect the data is 

often available from the data generator and is invaluable in documenting the suitability of the 

data for project use. The encouragement by the Administrator to apply peer review to a wider set 

of information increases the potential reliability of the data.

Data Accepted by the Courts; the Daubert Rule

When considering data for a source involving a litigated court case, reference to the Daubert 

Rules of evidence should be made. Daubert is one of the first of several rulings by the United 

States Supreme Court giving guidance on the admissibility of scientific theory or evidence 

(Brilis, Worthington, and Wait 2000). It is usually used in conjunction with Rule 702 of the 

Federal Rules of Evidence.

Scientific expert testimony (including data) submitted for testimony first has to be admitted by

the trial judge, who acts as a gatekeeper in assuring that the expert testimony truly proceeds from 

documented scientific knowledge. The trial judge uses relevance and reliability (the Daubert 

factors) as well as Federal Rule 702 to make the determination to admit expert testimony as 

evidence.

It follows that when data submitted for evidence are considered for possible use in another 

context, there is some sense that the data are reliable because they meet the Daubert and 702 

Rules. The data does, however, needs to be evaluated against the acceptance criteria for the 

proposed project because data accepted for one purpose may not be suitable for another.

It is axiomatic that basic QA principles involving the generation of the data have been achieved 

as it is not possible to meaningfully test hypotheses using unreliable data. It also follows that the 

data should be reliable because they have been peer reviewed.

The Daubert factors would also likely apply to novel situations where a standard approach to 

performing the operation does not exist or is incomplete in documentation. Attention should be 

given to the overlying charge that forms the basis for data inclusion for Agency use “to assure 
QAPP Handbook 62 December 2012 Agency Review Draft

that Agency decisions are supported by data of adequate quantity, quality, objectivity, and utility 

for their intended purpose” (see Clause 4 of the Internal Standard). Note that “intended purpose” 

is key to the establishment of clear objectives for the project, the associated measurements, and 

their associated quality control criteria.

However, being accepted for evidence does not automatically imply they are acceptable for other 

uses. The presiding judge has acted as a “gate-keeper” in allowing these data as relevant and 

acceptable for consideration for this particular case. Consideration and evaluation of whether the 

data meet the acceptance criteria established by the potential data user must still be determined 

and documented. 

3.4.2 Quantitative Comparisons to Acceptance Criteria

The QAPP should describe how results obtained from the project or task will be reconciled with 

the requirements defined by the data user or decision-maker. The proposed methods to analyze 

the data to identify possible anomalies or departures from assumptions established in the 

planning phase of data collection should be included in the QAPP. These should be determined 

prior to data collection to ensure they are driven by project needs. The QAPP should describe 

how reconciliation with user requirements will be documented, issues will be resolved, and how 

limitations on the use of the data will be reported to decision makers. 

Appendix B contains supporting information on DQIs for which MQOs can be set as acceptance 

criteria for including existing data in a project. The DQIs that may be able to be quantitatively

assessed are:

 precision;

 bias;

 accuracy (the combination of precision and bias);

 completeness; and

 sensitivity.

For each of these DQIs, the QAPP may include a plan for assessing project data against the 

acceptance criteria set during project planning and documented within the QAPP. The discussion 

of comparison of project data to acceptance criteria may include the following aspects:

 when in the project life cycle these comparisons will occur;

 who will be responsible for these comparisons;

 how the comparisons to MQOs will be documented; and

 what action will be taken if the project data do not meet the MQOs.

Statistical comparisons are very useful and are discussed in EPA 2006b which has examples and 

instructions on how to perform statistical tests. For the success of any project, team members 

should be encouraged to contribute freely in the overall evaluation of the project’s data. 

There are, however, many instances where comparisons to acceptance criteria for MQOs cannot 

be achieved. 
QAPP Handbook 63 December 2012 Agency Review Draft

This is usually because the published dataset only gives the final DQOs and not the specific 

MQOs. In such cases a note or caveat should accompany the description in the QAPP that this 

aspect of the reliability of the data could not be fully confirmed. 

3.4.3 Assessments to Other Criteria

Beyond the DQIs and MQOs defined for the project, there may be other criteria against which 

the use of data should be assessed. Examples of other aspects that may be included in such 

assessments include the following:

 cost or resource constraints for incorporating existing data;

 the timeframe in which the data will be available;

 access to the physical site of study; and

 environmental justice concerns.

The timing of these types of assessments may be mentioned in the QAPP (e.g., during readiness 

review, after the first data are incorporated, or at the conclusion of data analyses), as well as 

responsibility for conducting them and the process for documenting their conclusions.

3.4.4 Interim Assessments of Data Quality

It may be appropriate to conduct interim assessments as part of an iterative sequence of 

determining the suitability of existing data. The depth to which this can be done depends on the 

acceptance criteria established for the project. It is sometimes used to show why a certain data 

set should not be considered for a particular project. Possible indicators of data quality issues 

include:

 field sampling practices that potentially impact data quality or reliability;

 anomalous events potentially influencing data quality;

 extreme weather conditions; and 

 an intuitive sense that something isn’t right with the data.

Of course, the latter indicator is subjective and should be discussed with the entire project team. 

Its inclusion here is to act as a warning signal that further investigation into the background of 

the existing data is warranted. Interim assessments may also be combined with any results from 

the comparisons to acceptance criteria (Section 3.4.2).

3.4.5 Evaluation of Unconventional Measurements

Unconventional methods can be thought of as any method for which there is not an already 

existing standard that has been widely accepted and validated. If data or measurements from 

unconventional methods are used for a project, the QAPP should include scientifically defensible 

documentation why these procedures are appropriate for this project.

The quality of environmental data obtained via unconventional methods should clearly support 

the intended use for the project and, when possible, quantitative acceptance criteria for selected 

DQIs should be developed. 
QAPP Handbook 64 December 2012 Agency Review Draft

If qualitative acceptance criteria are constructed, then a clear, detailed, explanation of why these 

measurements are used is needed. Use of data from unconventional methods will come under 

increased scrutiny by reviewers, hence the need for detailed explanations. 

3.4.6 Evaluation of Unconventional Monitoring Projects

As with unconventional laboratory methods, it is especially important that nontraditional 

monitoring methods be well documented in the QAPP. Along with full descriptions of the 

methods, acceptance criteria for qualitative indicators will be needed. The use of the data 

generated from any nontraditional methods can be difficult unless documentation of why the 

proposed data should be used is included in the QAPP.

3.5 REVIEW, EVALUATION OF USABILITY, AND REPORTING 

REQUIREMENTS (ACT)

The final part of the paradigm concerns the actual data; are they valid and can they be used for 

the intended purpose. The evaluation of usability must include any perceived limitations on the 

use of the data and with the reconciliation with project requirements is a small discussion of 

statistical significance. This section offers guidance on Standards Clause 7.9, Data Review, 

Verification and Validation, and Data Usability Reporting, and Annex B3.6, Review, Evaluation 

of Usability, and Reporting Requirements.

The elements in this section address project checks to see if the data or product obtained will 

conform to the project’s objectives and to estimate the effect of any deviations. These activities 

may be performed throughout the project to inform mid-course corrections and/or at the 

completion of the project.

3.5.1 Data Verification and Validation Targets and Methods

For existing data it is not common for verification and validation details to be found and recourse 

has to be made to the provenance of the data. If the generator of the data has a known quality

system, there is some assurance that the data have been verified and validated. This is especially 

true for data generated by or for the EPA as the Agency’s quality system has verification and 

validation as an integral part of the system. Data from organizations with less documented 

quality systems may give sufficient cause for concern that caveats should be attached to any 

conclusions reached from the data.

3.5.2 Quantitative and Qualitative Evaluations of Usability

Usability of existing data depends on the objectives of the project and the extent to which the 

existing data meet quantitative and qualitative acceptance criteria. All existing data have use, but 

the strength of the conclusions reached by analysis of the data varies directly with the 

representativeness of the data with respect to the project’s objectives. Based on information 

gleaned through qualitative and quantitative comparisons to acceptance criteria, and 

reconciliation with project requirements, the project team considers the degree to which the data 

meet project quality objectives. This requires the careful discussion of where the data are

inadequate or relatively weak in desired quality. This might be as simple as a precision estimate, 

but could be as much as an unintended change of population coverage due to data collection 

methodology.
QAPP Handbook 65 December 2012 Agency Review Draft

Note that this differs from comparisons to acceptance criteria in that here project level objectives 

are being considered, previously only the specific MQOs were considered. A typical example 

would be to see if the estimated mean level of contaminant (one of the project objectives) can be 

estimated to within 10 ppb (another project level objective) with 95% certainty (the final project 

objective). 

The use of statistical comparisons to quantitative acceptance criteria is encouraged, but with a 

warning that any degree of statistical significance must be tempered by the extent to which the 

existing data deviates from a truly random sample. The statistical analysis can be used to “weigh 

the evidence” by making comparative conclusions instead of actual levels of significance. Data 

Quality Assessment: Statistical Methods for Practitioners, EPA QA/G-9S, (EPA 2006b) contains 

many commonly used statistical procedures and has been written for the non-statistician.

Comparisons to qualitative acceptance criteria such as representativeness and comparability are

by definition, subjective, and can be affected by the unintentional bias of the project team. 

Qualifiers and caveats as to interpretation of the results are to be expected (see also Section 

3.5.3). It is advised that a summary of the project team’s collective reasoning be included in the 

conclusions from the analysis of the data. 

3.5.3 Potential Limitations on Data Interpretation

The QAPP should address the action to be taken when project-required MQOs (acceptance 

criteria) are not achieved or for other reasons existing data may not be immediately usable with 

respect to the project objectives. In addition to the potential data usability issues, there are other 

considerations that might limit the interpretation of the data such as: 

 information gaps regarding data collection methodology;

 information gaps regarding QA procedures used during data collection;

 use of nonstandard analysis methods;

 undocumented gaps in supporting metadata;

 inclusion of potentially anomalous data not documented as outliers;

 sample support (e.g., different amount of media collected per sample, or samples 

representing different sized areas); and 

 the use of a judgmental sampling design. 

The QAPP should state that any potential limitations of data interpretation should be documented 

and included in the final project report and included in the metadata stored with the project data.

The use of footnotes and separate appendices to caveat weaknesses in interpretation of data is 

discouraged as these tend to be omitted when project reports are sent to outside parties. The 

potential limitations should be in the text of the summary report.

3.5.4 Reconciliation with Project Requirements

Considering how the data will be used to answer the main study questions is an important step in 

project planning, and should be documented in the QAPP. 
QAPP Handbook 66 December 2012 Agency Review Draft

Without consideration of this, it is possible to compile data of high quality that actually cannot 

be used to answer questions because no statistical methods are available to support the project 

needs. The QAPP should describe how the data will be summarized or analyzed (e.g., qualitative 

analysis, descriptive, or inferential statistics) to meet project objectives. If descriptive statistics 

are proposed, state what tables, plots, and/or statistics (e.g., mean, median, standard error, and 

minimum and maximum values) will be used to summarize the data. If an inferential method is 

proposed, indicate whether the method will be a hypothesis test, confidence interval, or 

confidence limit and describe how the method will be performed. Specifically, the QAPP should 

briefly discuss the five steps of the DQA process (see EPA 2006b):

 a review of the project’s objectives to assure that they are still applicable and a review of 

the sampling design and data collection documentation for consistency with the project 

objectives noting any potential discrepancies;

 a preliminary data review of QA reports (when possible) for the validation of data, 

calculation of basic statistics, and generation of data graphs to learn about the data 

structure and to identify patterns, relationships, or potential anomalies;

 selection of the appropriate statistical procedures for summarizing and analyzing the data, 

based on the review of the performance and acceptance criteria associated with the 

project’s objectives, the sampling design, the preliminary data review, and identification 

of the key underlying assumptions associated with the statistical tests;

 verification that those underlying assumptions most likely hold, or whether departures 

could be acceptable, given the actual data and other information about the study; and

 determination of conclusions from the data by performing the calculations pertinent to the 

statistical test, and documentation of the conclusions to be drawn as a result of these 

calculations. If the design is to be used again, there should be an evaluation of the 

performance of the sampling design.

Although the steps of the DQA will not be implemented until data compilation is complete, 

planning for DQA including selecting likely statistical methods for analyzing the data and 

understanding the underlying assumptions of those methods is vital to do prior to data 

compilation to avoid missteps. Quantitative (statistical) evaluations of the data can provide 

strong evidence in support of (or to refute) project hypotheses.

When considering using published data for purposes other than what it was collected for, 

attention should be given to the practical significance otherwise misinterpretation of importance 

may ensue. This is particularly true when considering the use of data concerning a popular area 

for research or study. It is possible that the significant result was published, while all the nonsignificant results were deemed not suitable for publication.

Statistical significance is a concept based on the weight of evidence that a hypothesis is valid. It 

is never possible to have perfect knowledge about a population being studied, but it is possible to 

learn enough about it to be able to say with confidence that a particular hypothesis concerning 

that population cannot be true. However, one should be very careful not to allow statistics to 

dictate decisions without recourse to common sense. In particular, as more data are collected, it 

becomes easier to achieve statistical significance. 
QAPP Handbook 67 December 2012 Agency Review Draft

The concern is that at some point it may be possible to determine statistical significance at levels 

that are not of practical significance. This can be illustrated through the following example:

Based on operations at an industrial plant, and their waste-release permit, it is expected 

that the pH of water leaving the plant will be 5.9. The releases are monitored by weekly 

collections and each week these data are combined with all previous data and the average 

pH is compared to 5.9. After the first few months, the average release pH is 5.88, which 

is not statistically significantly different from 5.9 and the conclusion of no real difference 

justified. After several years have elapsed the average release pH is 5.8996 and this is 

statistically significantly different from the permitted value of 5.9, but is a conclusion of a 

real difference justified? This is a case where having so much data allows the reviewer to 

identify very small differences from the expected level, but the statistically significant 

result may very well not have any practical significance (in this case a difference in pH of

0.0004, which is barely measurable). 

While statistics provide a strong and essential tool for environmental decision making, the 

science of statistics is not a substitute for common sense and can lead to bad decisions if not 

tempered with practicality. This is particularly true with publications that tend to accept only

papers that show a “statistically significant result” (often using 5% or even 1% as the level of 

significance). It is relatively rare that the publication editor has required the author to include a 

discussion of the practical meaning of the statistical significance using costs, benefits, and 

analysis of the magnitude of the observed effects. The QAPP should take care to specify the 

conditions under which statistical significance may not have practical meaning.

It is worth noting that “statistical significance” should not be the sole indicator of importance of 

a result or conclusion. This was discussed by the U.S. Supreme Court in Matrixx Initiatives, Inc 

v. Siracusano (March 22, 2011) (Ziliak 2011). While the case involved security law, the 

defendant (Matrixx) tried to suggest that its line of demarcation on whether to release data was 

based on statistical significance. The Court disagreed unanimously. They said that the presence 

or absence of statistical significance is not the key factor as to whether an adverse effect is 

material. Justice Sotomayor wrote the opinion noting, “A lack of statistically significant data 

does not mean that medical experts have no reliable basis for inferring a causal link between a

drug and adverse events”. Statistical significance is only part of the quantitative aspect in the 

interpretation of a project’s results.

The QAPP should also identify data reporting requirements, including data reduction procedures 

specific to the project and applicable calculations and equations. In addition, data storage 

requirements for both hard copy and electronic data, as well as software requirements for the 

analysis should be specified.

3.5.5 Reports to Management

The QAPP should identify the frequency and distribution of reports issued to inform 

management (EPA or otherwise) of the project status, including results of periodic data-quality

assessments, significant QA problems, and recommended solutions. 

Periodic QA management reports are intended to ensure that managers and stakeholders are kept 

informed of project status and results of all QA assessments. 
QAPP Handbook 68 December 2012 Agency Review Draft

Efficient communication of project status and problems will allow project managers to 

implement timely and effective corrective actions so that data produced for the project can meet 

the project’s objectives.

The QAPP should describe the content of each QA management report to be generated for the 

project, including an evaluation of measurement error (when possible) as determined from the 

assessments. Assessment checklists, reports, requests for corrective action letters, and the 

corrective response letters should be included as attachments to or referenced in the QA 

management reports.

The following issues may be included in QA management reports:

 summary of project QA program activities including tables, charts, graphs depicting 

results;

 status of project and schedule delays;

 deviations from the approved QAPP and approved amendments to the QAPP;

 required corrective actions and implementation; and

 limitations on the use of measurement data generated.

Although these issues listed may be addressed in QA management reports, they should also be 

included in the QA section of the final project report. The final project report should, at a 

minimum, give a reconciliation of project data with project objectives, data summary (including 

tables, charts, and graphs), a summary of major problems encountered and their resolution, and 

any additional data quality concerns together with conclusions and recommendations. As many 

Agency reports are being read by an ever increasing audience, it is recommended that sufficient 

detail and explanation is presented in the final report to allow for an independent assessment of 

whether the existing data used in the project is suitable for reaching the project’s objectives.
QAPP Handbook 69 December 2012 Agency Review Draft

CHAPTER 4

QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, AND USE OF MODELS

4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS

The use of models is becoming an increasingly important part of the Agency’s investigation into 

environmental phenomena and Agency Quality Policy CIO 2106.0 mandates the use of a QAPP. 

The QAPP has some elements very similar to those for QAPPs for new or existing data, but there 

are important differences to allow effective model development and use. This Chapter offers 

advice on Standards Clause 7.6, Use of Information Technology Methods and Sources, and 

Annex B3.4, Data Acquisition. 

This chapter focuses on QAPPs for developing, modifying, applying, and evaluating models, and 

for project use of data from models. The QAPP, using the elements described in this chapter, 

should provide the project team with a clear path for determining if use of models will achieve 

the project objectives or DQOs.

Probably the most important part of the QAPP is in the planning for obtaining the data or 

information because an error at this stage may be difficult to rectify once the project is 

underway. One of the key outputs of the systematic planning process is the construction of a 

conceptual model that describes the scientific and engineering process under investigation. The 

conceptual model is an important tool for organizing information about the current state of 

knowledge and understanding of the project, as well as for documenting key theoretical and 

practical assumptions underlying the data and information collection. In some modeling projects 

the conceptual model is the model itself, but in others the model being investigated may be only 

part of the complete conceptual model. Further details on systematic planning are to be found in 

Annex C of the Standards and EPA 2006a.

The use of models adds many nuances to traditional EPA QAPPs. The QAPP is meant to provide 

a description of how the project team will ensure that the project attains a level of quality 

commensurate with its importance. The project team should document all aspects of the project 

that could impact project quality. The QAPP is not meant to be burdensome, and any sections of 

this chapter that do not pertain to a particular project involving the development, modification, or 

use of models should simply present why those sections do not. The ultimate purposes of the 

QAPP are for the project team to ensure that using the data, model, or both result in achieving 

the project objectives, and to provide documentation and defensibility for the quality of the 

project results.

EPA uses a wide range of probabilistic, fate and transport, and other types of models to inform 

decisions that support its mission of protecting human health and safeguarding the natural 

environment – air, water, and land – upon which life depends. These models include atmospheric 

and indoor air models, ground-water and surface-water models, multimedia models, chemicalequilibrium models, exposure models, toxicokinetic models, risk-assessment models, and 

economic models. These models range from simple to complex and may employ a combination 

of scientific, economic, socio-economic, or other types of data. As defined in this guidance, a 

model is an application tool, while a model framework is the system of governing equations.
QAPP Handbook 70 December 2012 Agency Review Draft

As models become increasingly significant in decision making, it is important that the model 

development, modification, use and evaluation processes conform to protocols or standards that 

help ensure the utility, documentation, and defensibility of the models and their outputs for 

decision making. EPA’s Council for Regulatory Environmental Modeling (CREM) is a valuable 

resource for modelers with Guidance on the Development, Evaluation, and Application of 

Environmental Models (EPA 2009b) and self-study training modules at www.epa.gov/CREM.

The following figure (Figure 6, The Modeling Process) [EPA 2009b, page 61, figure D.1.1]

shows model development, evaluation, and application, and the relationship between them, 

including the use of data, data quality assessment and peer review. Note the order in the figure, 

where evaluation is central to both development and application. The development and use of 

mathematical and electronic models (IT products) should be described in a QAPP and meet the 

applicable requirements of the Internal and External Standards. Further information is provided 

in this chapter; the elements are described in more detail in this chapter and organized by the life 

cycle of the project according to the Plan-Do-Check-Act paradigm.

QAPPs should, at a minimum, address all elements required by the Internal and External 

Standards as detailed in this guidance. Elements can be addressed with a simple statement of 

why the information is not relevant, as mentioned previously, or with a cross-reference to 

another approved document in which the information appears. It is acceptable to add sections to 

a QAPP beyond those identified. For example, if a project involves heavy use of computers run 

in parallel to accomplish runs of complex models, the computer requirements might be added as 

a separate element in the QAPP. 

If the project must first set objectives and define what is to be accomplished by the model, it may

be most appropriate to develop the QAPP in a phased manner (see Chapter 1, Section 1.5). The 

first stage might include all of the project planning and project quality objectives, and the latter 

stages might include model development or model selection and setting a schedule for the 

appropriate assessments to ensure the model is operating as expected. If a phased approach is 

used, the project must not move beyond the activities addressed in the QAPP until the next phase 

of the QAPP is developed and approved. 

This chapter focuses on QAPPs for modeling projects with the elements are arranged according 

to the Plan-Do-Check-Act project life cycle. 

4.2 PROJECT MANAGEMENT (PLAN)

Planning, the first part of the construction of a QAPP, deals mainly with the management 

aspects of the project. Most of the structure is simple good management practices, the only QArelated part being the establishment of data or project quality objectives and measurement 

performance criteria (section 2.2.6). The section offers guidance on Standards Clause 7.5, 

QAPP, Standards Clause 7.7, Systematic Planning, Standards Clause 7.10, Documents and 

Records Management, Annex B3.3 Project Management, and Annex B3.4 Data Acquisition as it 

applies to models. 

The elements in this section address the format and disposition of the QAPP, project 

administrative functions, project information, and goals. These elements document the backbone 

of the project planning process and lay the groundwork for the more technical elements.
QAPP Handbook 71 December 2012 Agency Review Draft

 Figure 6. The Modeling Process
QAPP Handbook 72 December 2012 Agency Review Draft

The QAPP must describe the project adequately and the elements that can address the basic 

project management and objectives of the work include:

 title, version number, and approval/sign-off sheets;

 document format and table of contents;

 distribution list;

 project organization and schedule;

 project/problem background and description;

 data or project quality objectives and measurement performance criteria; 

 special training requirements/certification; and

 documentation and records requirements.

4.2.1 Title, Version, and Approval/Sign-Off 

Each QAPP should include a page with the title of the project and the name of the organizations 

involved in various aspects of that project. The version of the QAPP should also be clearly 

identified along with the title. It is acceptable to create separate title pages and signature pages, 

as long as the document title, version number, and date appear on the signature page. The names, 

titles, signatures, and signature dates of those approving the plan are also placed on this page. 

Individuals responsible for approving the QAPP may include the organization’s Technical 

Project Manager and QA Manager, and the EPA Project Manager and QA Manager. Their 

signatures indicate both their approval of the plan and commitment to follow the procedures 

noted. Other key personnel that may sign the plan are upper management, other QA officers, 

prime contractors, and subcontractors. The QA Manager will also determine if digital 

(electronic) signatures are acceptable for the approval of the QAPP. This approval information is 

typically the first page of the QAPP. Depending on the organization’s administrative policy, 

QAPP approval could also be in a separate memorandum. The signature dates indicate the 

earliest date when the environmental data operations portion of the project can start (i.e., its 

effective date).

In addition to the title, version number, and approval signatures, it is important to include a 

revision history. Each time the QAPP is revised, as approved by the QA Manager, the version 

number should be updated and the revision history should be amended to include a brief 

summary of the change and date. 

4.2.2 Document Format and Table of Contents

The QAPP should be organized such that it meets the project’s needs, can be reviewed 

efficiently, and meets the document control requirements of the QMS under which it is 

developed. A document control format, such as the example shown in Figure 7, may be used to 

support QAPP development, or a footnote created for each page to show revision status. 
QAPP Handbook 73 December 2012 Agency Review Draft

Figure 7. Document control format example

The Table of Contents will generally list QAPP elements, as well as any tables, figures, reference

section, and appendices necessary to the project. If the QAPP author prefers to organize the plan 

differently than how the elements are organized in this Handbook a table may be inserted here to 

cross-reference where the information for each element may be found to simplify review. SOPs 

may be included as appendices to the QAPP. Depending on the type of project, analytical 

research protocols, or data management procedures may be attached. If SOPs or other data 

gathering, data analysis, or evaluation protocols are not documented in, or attached to the QAPP, 

they must be available through some other means. The location of the SOPs referenced in the 

QAPP should be accessible to both QAPP users and QAPP reviewers.

4.2.3 Distribution List

The distribution list identifies all individuals who should get a copy of the approved QAPP, 

either in hard copy or electronic format, as well as any subsequent revisions. Key personnel 

responsible for project implementation and funding, and who should have the currently approved 

QAPP, should be listed with their project titles or positions, organization names, email addresses, 

and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will 

need access to it, the distribution list also serves as an easy reference of who needs to be alerted 

and provided with a revised version of the QAPP in the case that modifications are necessary. 

Some organizations choose to provide the distribution list on the title or approval page, others 

elect to include this list in the project organization section when listing key personnel and their 

contact information.

4.2.4 Project Organization and Schedule

It is important that roles and responsibilities are well defined prior to initiating project activities. 

Those individuals involved with the major aspects of the project are named in this element along 

with their project responsibilities. For example, the staff responsible for maintaining, updating, 

and overseeing implementation of the QAPP would be named here. The personnel included in 

this element should include the lead scientists, researchers, modelers, consultants, QA personnel, 

and contact information for involved external organizations (contractors and sub-contractors). If 

the actual personnel cannot be initially identified, then the position description of that person’s 

function should be given. At a later date, after personnel have been identified, amendments to the 

QAPP may be made (at the discretion of the QA Manager).

An organizational chart or table can be very helpful, and should be included if appropriate. It is 

also helpful to indicate lines of communication among individuals or groups, and this can be 

shown easily on an organizational chart or an organizational network diagram. 

Project Name/# __________________

Section # _______________________

Revision # ______________________

Date ___________________________

Page ___________ of ____________
QAPP Handbook 74 December 2012 Agency Review Draft

While a single individual may have more than one responsibility in a project, the project should 

be organized such that any person having QA responsibilities is independent of those generating 

and using project information. If this is not possible, an acceptable alternative method of 

ensuring effective QA review should be specified in the QAPP.

The level of detail included in the schedule is left to the discretion of the QAPP authors. It may

be beneficial to have a very detailed and strongly stated schedule for the project to follow. This 

increases the a risk of requiring QAPP revisions if the schedule needs to be changed during the 

lifetime of the project unless the QAPP specifically indicates that such changes need not cause a 

formal revision of the QAPP. It is useful to include critical points in the project such as expected 

date of QAPP approval, sub-section start and end points, or dates modeling subroutines need to 

be completed. When creating the schedule, allowance should be made for potential delays and 

general inefficiencies inherent in any project. If the project includes regulatory or courtmandated deadlines, these should be highlighted to ensure their importance is noted. For projects 

in which deadlines or milestones are not well defined, a more generalized work schedule can be 

formulated.

The project management elements of a QAPP for modeling activities are parallel to those for a 

QAPP for a project based on newly collected data or on existing data, as described in Chapters 2 

and 3, respectively. These chapters can be helpful for projects combining activities.

4.2.5 Project Background, Overview, and Intended Use of the Model

For modeling projects, more effort is put into this, the planning aspect of project development, 

than any of the other parts of Plan-Do-Check-Act paradigm. This stems from the fact that if 

insufficient attention is given to planning, potentially erroneous results may occur (a variation of 

“garbage-in, garbage-out”).

This overview should give the reader an understanding of the problem to be solved, along with 

any pertinent background information for the project to provide a historical and scientific 

perspective for future projects. It describes why the project will be done and what goals the 

project intends to accomplish. The general project goals stated here form the foundation for the 

entire study. Equally important, the development and documentation of this element will help to 

ensure that all project team members clearly understand and agree on the underlying purpose of 

the project, increasing the likelihood that the project design will address and accomplish that 

purpose. Included in this discussion should be a summary of any previously approved QAPPs 

relevant to the project or study. This should include both those specific to modeling and those 

involving the collection or use of data (see also section 4.2.6).

4.2.5.1 QAPP Requirements for Model Development/Modification

This section expands on the demands of Standard B3.4.4.1, QAPP Requirements for Model 

Development/Modification and integrates specific advice with the Council for Regulatory 

Environmental Modeling “Guidance on the Development, Evaluation, and Application of 

Environmental Models”.

After the specification of the project objective and type of model needed to address the problems 

associated with the project objective is the development of a suitable model framework. 
QAPP Handbook 75 December 2012 Agency Review Draft

“A model framework is a formal mathematical specification of the concepts, procedures, and 

behaviors underlying the system, object, or process relevant to the problem of interest, usually 

translated into computer software” (EPA 2009b). The structure of the model should be clearly

stated especially whether it is a stochastic model or a deterministic model as this is of crucial 

importance in the interpretation of model output. A stochastic model (one founded on 

probabilistic principles involving random events or fluctuations) may result in an output where 

the results must be interpreted by a series of probabilistic (statistical) statements. A deterministic 

(sometimes called a mechanistic) model will usually not carry a probabilistic statement.

“For mechanistic modeling of common environmental problems, one or more suitable model 

frameworks may exist. Many existing model frameworks in the public domain can be used in 

environmental assessments. Several institutions, including EPA, develop and maintain these 

model frameworks on an ongoing basis. Ideally, more than one model framework will meet the 

project needs, and the project team can select the best model for the specified problem. 

Sometimes no model frameworks are appropriate for the task, and the team will develop a new 

model framework or modify an existing framework to include the additional capabilities needed 

to address the project needs” (EPA 2009b).

“During problem specification, the project team defines the regulatory or research objectives, 

the type and scope of model best suited to meet those objectives, the model performance and 

data criteria, the model’s domain of applicability, and any programmatic constraints. These 

considerations provide the basis for developing a conceptual model, which depicts or describes 

the most important behaviors of the system, object, or process relevant to the problem of interest. 

Problem specification and the resulting conceptual model define the modeling needs that the 

project team can then determine whether an existing model can be used to meet those needs or 

whether a new model should be developed.”(EPA 2009b)

Of importance to model modification is consideration of the original model origin and purpose. 

The assumptions used in constructing the original model may be very different than those being 

used now. The effectiveness of new models built on existing models using different assumptions 

can lead to erroneous conclusions.

For models to be developed, modified, and evaluated, objectives and performance criteria should 

be defined for both the inputs and outputs of the models.

Problem specification and identification of model purpose and scope

From Chapter 3.2, EPA 2009b: “Problem specification, culminating in development of the 

conceptual model, involves an iterative, collaborative effort among model developers, intended 

users, and decision-makers to specify all aspects of the problem that will inform subsequent 

selection or development of a model framework. (Manno et al. 2008).” 

There are three important planning areas the QAPP needs to cover: objectives, important 

functions, and model scope.
QAPP Handbook 76 December 2012 Agency Review Draft

Objectives:

 the purpose or general objectives of the project/model, and the expected applicability or 

scope of the model and this includes general statements such as what questions the model 

is designed to answer; and

 sufficient background information to provide a historical and scientific perspective for 

context setting for this particular project, and to act as a basis for future related projects.

Important Functions:

 identification of the key functions or outcomes the software must perform, key decisions

to be made, actions that need to be taken, and an outline of the outcomes expected from

the information obtained from modeling activities.

Model Scope:

 the spatial and temporal limits for the model including any key assumptions on model 

applicability to the particular and any limitations on model applicability for this modeling 

process.

Model development, modification, or selection process

For model development a clear statement of the basis for the model including its mathematical 

structure, how software will be used in the development of the new model, system requirements, 

and procedures for software version control is needed. It is essential that the QAPP addresses:

 the theory that forms the basis for the model, the mathematical algorithms, and 

approaches used in executing the model computations; 

 the existing software to be used in development of a new model, how it will be used, why 

it was selected (subject to the requirements of CIO 2123.0), and model scope (e.g., 

spatial, temporal and process detail); 

 the computer hardware and operating system requirements for the software application 

together with the model code development and verification plans, and the most important 

functions that the software application must perform; and

 how the model structure reflects all the relevant inputs to the conceptual model and the 

expected applicability and limitations of model use. 

4.2.5.2 QAPP Requirements for Model Application

This section expands on the demands of Standard B3.4.4.2, QAPP Requirements for Model 

Application and integrates specific advice with the Council for Regulatory Environmental 

Modeling “Guidance on the Development, Evaluation, and Application of Environmental 

Models.”
QAPP Handbook 77 December 2012 Agency Review Draft

“The scope (i.e., spatial, temporal and process detail) of models that can be used for a particular 

application can range from very simple to very complex depending on the problem specification 

and data availability, among other factors. When different types of models may be appropriate 

for solving different problems, a graded approach should be used to select or develop models that 

will provide the scope, rigor, and complexity appropriate to the intended use of and confidence 

needed in the results.” (EPA 2009b) 

Many of the QAPP requirements for model applications are similar to those needed for model 

identification, and should be noted in the QAPP. The QAPP should document the “how and 

why” the model will be useful in meeting the project’s objectives. An effective QAPP should be 

such that a reviewer or interested reader should be able to understand the importance or context 

of the project. The QAPP has to document the specifications for model application.

Problem Selection, Model Purpose, and Model Selection

There are two key areas the QAPP needs to cover: the basis for selection of model, and purpose 

of model application and analysis.

Basis for Model Selection:

 the rationale for choosing this particular model from other competing models; “In all 

cases, the QAPP should clearly document for the selected model, why and how the model 

can and will be used, the spatial extent and resolution, and the temporal extent (length of 

the modeling period)” (EPA 2009b). 

Purpose of Model Application:

 the research or regulatory objective of the application or the purpose for applying the 

model for this particular project.

Model Application

For model application the two most important areas the QAPP must define are scenarios, and 

results.

Scenarios:

 it is rare that one model can fit multiple scenarios and so it important that the QAPP 

accurately defines the probable simulation scenarios to which this model will be applied. 

This documentation will help prevent misunderstandings as to what the model applies to.

Modeling Analysis and Results

 the modeling analysis and discussion of results is very important to establishing the 

validity of this particular application. Documentation of these results is an essential part 

of model development and application.
QAPP Handbook 78 December 2012 Agency Review Draft

4.2.6 Data/Project Quality Objectives and Acceptance Criteria

In many cases the dataset being considered for use is incomplete in some aspect. Such datasets 

are often “repaired” using imputation techniques such as nearest neighbor averaging or estimated 

parameter forecasting. It is essential that the QAPP should document both the extent of 

imputation and the methodology used.

4.2.6.1 Data Requirements for Model Development or Modification

It is acknowledged that when considering existing data for potential use in a modeling project, 

quantitative acceptance criteria may be difficult to construct. However, some concept of what the 

acceptable range of acceptance criteria needs to be developed by the project team. Reliance on 

broad statements such as “we’ll use the best we can find” or “whatever is in the literature” 

should be discouraged. If the quantitative criteria are poorly defined, then an iterative approach 

may be appropriate where the characteristics of one existing data set are compared to those of 

another until the “best” is selected. It is advised that the acceptance criteria be phrased in a 

comparative fashion by selecting an important DQI and then addressing the MQO for that 

indicator. For example, “the dataset having the most precise estimate of variance will be 

selected”, the desired indicator, precision, is the acceptance criterion for the different datasets 

under consideration.

Qualitative acceptance criteria (for example, representativeness, or comparability) may be 

difficult to describe, but need to be couched in terms that clearly indicate why one set of existing 

data may be preferred to another. For example, “the project’s objectives require the cohort of 

data be exposed to defined environmental conditions for a period not exceeding 5 years”, the 

selected indicator, comparability, being linked to the objectives of the project.

When an existing dataset lacks information on MQOs, careful attention should be given to the 

overall DQOs of this dataset, because the set of MQOs chosen for the key indicators directly 

affect the DQOs. A very “tight” achievement of the DQOs would be a good augury of the quality

of the dataset, a “looser” achievement, less so. 

While historically considerable attention has been directed to the principal DQIs of precision,

bias, representativeness, comparability, completeness, and sensitivity, it is really

representativeness that is probably the single most important indicator of data quality. 

Representativeness is a qualitative measure of the degree to which data accurately and precisely 

represent a characteristic of a population. If the data used in the model fail to be representative of 

what was intended, no amount of mathematical manipulation or adjustment can make the results 

valid. For modelers, the DQIs of calibration technique, uncertainty analysis, sensitivity analysis, 

and qualitative comparisons to other studies become more important. The QAPP has to set both 

quantitative and qualitative criteria with what is likely to be available from data analysis or 

available through qualitative comparisons. In either case, acceptance criteria need to be 

documented (see also section 3.4 of this guidance).
QAPP Handbook 79 December 2012 Agency Review Draft

4.2.6.2 Criteria for Acceptance for Model Application 

The criteria for model application are similar to those encountered for model development or 

modification and so the opening remarks of section 4.2.6.1 apply to model application as well. 

In this element, the QAPP should indicate how “good” the input data and the model need be to 

answer the question, resolve the problem, or support the decision at the level of quality desired. 

The intended use of the model should be clearly stated and if used to extrapolate beyond the 

range of data entered into the model, the appropriate caveats should be included.

For each type of data or aspect of the model, indicate how accurate it needs to be and how to 

ensure the integrity of the information so that it can be relied upon and used for the intended 

purpose. Without making this clear, data or model outputs may be inadequate for final use, or 

conversely, more expensive to collect or produce than necessary because a higher standard of 

quality is targeted than is required for the project purposes. 

The QAPP should clearly state the criteria for acceptance of the model performance or model 

results. Acceptance criteria are the specific limits, standards, goodness-of-fit, or other criteria on 

which a model will be judged as properly calibrated (e.g., the percentage difference between 

reference data values from the field or laboratory and predicted results from the model). This 

includes the types of data and other information that will be necessary to acquire in order to 

determine that the model is properly calibrated (e.g., field data, laboratory data, and predictions 

from other accepted models [see Section 4.3.1]). In addition to addressing these questions when 

establishing acceptance criteria, the QAPP should document the likely consequences (e.g., 

incorrect decision making) of selecting data that do not satisfy one or more of these areas (e.g., 

are non-representative or are inaccurate), as well as procedures in place to minimize the 

likelihood of selecting such data. 

“If the team is evaluating multiple model frameworks, it may be useful to statistically compare 

the performance of these competing models individually and in aggregate with observational, 

field, or laboratory data.” (EPA 2009b) If that is the case, the QAPP should state how those 

comparisons will be conducted and evaluated. 

More detailed documentation of data acquisition and use procedures is described in Section 4.3.

4.2.7 Special Training Requirements and Certification

Special training or certifications are sometimes necessary for project personnel. For modeling 

projects, this may include training in the use of certain models, software, or programming 

languages; or the necessity for Confidential Business Information security clearance. The QAPP 

should also specify how this information will be documented, where the records will be kept, and 

indicate who is responsible for ensuring that these special training and certificate needs are met 

and documented.
QAPP Handbook 80 December 2012 Agency Review Draft

4.2.8 Documentation and Records Requirements

The QAPP should describe the process and responsibilities for ensuring that project personnel 

will receive the most recently approved project documents such as the QAPP, SOPs, and other 

documents used throughout the project operation. QAPP authors should discuss how these 

documents will be updated and how information regarding updates will be communicated. 

The QAPP should indicate where all project documents such as the QAPP or final report and 

records such as statutory requests and training records will be stored and for how long. Include 

backup procedures for any data stored electronically and cite the protocols for access to, retrieval 

from, and photocopying of information archives. In some cases retention, access to, and final 

disposition of some records may be regulated. In those cases, this element should address and 

comply with all relevant regulations. 

The QAPP should identify the information to be documented and can include:

 a description of the reporting format for model development/modification, evaluation and 

application, and for model inputs and model outputs;

 identification of other project records to be maintained, how/where the records will be 

stored, and the length of storage time; types of records may include:

o technical reviews, model tests, data quality assessments of output data and results, 

along with signatures of individuals performing review and approval,

o candidate model assessments for model selection, including references,

o actual input used and databases used,

o SOPs for model development and evaluation,

o pre- and post-software development information,

o spreadsheet data files containing monitoring data,

o copies of modeling reports, and

o model code, equations, and user’s guide (including information on the location of 

earlier versions).

The QAPP should indicate where all project documents (e.g., QAPP or final report) and records 

(e.g., statutory requests or training records) will be stored and for how long. The QAPP should 

identify any requirements set by statute or policy together with information on access, retention, 

and final disposition.

4.3 DATA ACQUISITION: MODEL DEVELOPMENT, MODIFICATION, AND USE

(DO)

For modeling projects there is less of a clear line between “plan” and “do” than for projects 

involving simply data. This section offers advice on the Standards Clause 2.1, Scope, Standards 

Clause 7.7, Systematic Planning, Annex B3.4, Data Acquisition, and Annex B4.4, Development 

and Use of Mathematical and Electronic Models. Some of the elements listed in the “plan” part 

could be shifted to the “do” part depending on the intent of the modeling project. For example, 

some of Data Requirements for Model Development or Modification (4.2.6.1) or Criteria for 

Acceptance for Model Application (4.2.6.2) could be placed here or even in the “check” section 

(4.4). The exact placement depends on the objectives and logistics of model development.
QAPP Handbook 81 December 2012 Agency Review Draft

The elements in this section address all aspects of data production and collection to ensure that 

appropriate methods for sampling, measurement and analysis, data production, data handling, 

and QC activities are employed and documented. The following QAPP elements describe the 

actual methods or methodology to be used for the collection, handling, and analyses of samples, 

and the management (i.e., compiling and handling) of the data.

The selection of data to be used in any modeling project is critical to meaningful interpretation of 

the results of the project. Knowing who originated (collected or was responsible for collecting) 

the proposed data can be very informative. Data originators might be EPA, states, Tribes, 

municipalities, potentially responsible parties (PRPs), or activist groups, for example. 

Confidence in the data may, to some small extent, be judged according to the data originator. 

That is not to say that data collected by states, for example, should always be accepted, or that 

data collected by PRPs, for example, should always be excluded. However, knowing who the 

data originator was may offer some information as to the purpose of the data collection and 

reflect on the possible impartiality of the lead author or source of the data.

If the data to be used comes from an EPA source, then the method of data collection, the results, 

and limitations on use of the results should be available in the data’s QAPP. Reference to the 

QAPP and its location should be made, and important extracts from that data QAPP included in 

the modeling QAPP. If the data source is outside EPA attention should be given to the issues 

raised in Chapter 3.3 of this Handbook (Data Acquisition, Existing Data).

If new data are to be generated for use in modeling, a separate QAPP for the data will probably 

be needed. If existing data are to be used, the information in Chapter 3, Section 3.4 and 3.5 

should be incorporated into the modeling QAPP.

The methods described in these elements should have been summarized earlier in Section 4.2.6. 

The purpose here is to provide detailed information on the methods. If the designated methods 

are well documented and are readily available to all project participants, citations are adequate; 

otherwise, detailed copies of the methods or SOPs should accompany the QAPP either in the text 

or as attachments.

4.3.1 Data Requirements for Model Input

For projects involving modeling, the data to be used in the model should meet specified 

objectives, as should the outputs of the model. The QAPP should define how the data quality 

acceptance criteria will be determined (for all information to be collected including information 

obtained from previous studies). The QAPP should also explain how the data acceptance criteria 

relate to the desired quality of model outputs.

4.3.1.1 QAPP Requirements for Model Development or Modification

For model development, Annex B3.4.4.1 indicates the QAPP must define:

 the data needs/inputs/sources;

 any boundary or initial condition specifications;

 data reporting requirements and format for model inputs/outputs; and

 the data quality objectives for model input and data used for model development.
QAPP Handbook 82 December 2012 Agency Review Draft

The QAPP should document the quality and quantity of data necessary for model development, 

modification, use, or evaluation. DQOs and specified acceptable range of uncertainty for the 

model input data should be defined as they “provide the specifications for model quality and 

associated checks. Well-defined DQOs guide the design of monitoring plans and the model 

development process (e.g., calibration and verification). The DQOs provide guidance on how to 

state data needs when limiting decision errors (false positives or false negatives) relative to a 

given decision. The DQOs should include a statement about the acceptable level of total 

uncertainty that will still enable model results to be used for the intended purpose. Uncertainty 

describes the lack of knowledge about models, parameters, constants, data, and beliefs. Defining 

the ranges of acceptable uncertainty  either qualitatively or quantitatively  helps project 

planners generate “specifications” for QA planning and partially determines the appropriate 

boundary or initial condition specifications and complexity for [model development].” (EPA 

2009b)

For developing or modifying models, any boundary or initial condition specifications should be 

defined. In applying a model or software, availability of data and knowledge of inputs needed is 

central. Data quality and quantity should be noted, along with procedures for excluding data.

4.3.1.2 QAPP Requirements for Model Application

For model application, Annex B3.4.4.2 gives the data requirements by focusing on data reporting 

requirements, format for model inputs and outputs, and discussion of issues regarding 

information on how data will be acquired and used in the project: 

 the need and intended use for each type of data or information to be acquired;

 selected source of the data and how it will be differentiated from other sources in the data 

management system; 

 how the data will be identified or acquired together with the basis for exclusion of 

competing datasets; 

 the method of determining the underlying quality and quantity of the data; and

 the criteria established for determining whether the level of quality for a given set of data 

is acceptable for use on the project.

4.3.1.3 DQIs for all Types of Modeling Projects

Some of the key criteria for individual data values, or a set of data values, include an 

examination of their DQIs with respect to potential use in modeling projects: 

Precision: Is the estimate of variability small enough to meet the uncertainty objectives of the 

modeling project? 

Bias: Would any characteristics of the dataset directly impact the model output (e.g., unduly high 

or low process rates)? For example, was bias in result analysis documented? Is there sufficient 

information to estimate and correct bias? If using data to develop probabilistic distributions, are 

there adequate data in the upper and lower extremes of the tails to allow for unbiased 

probabilistic estimates? 
QAPP Handbook 83 December 2012 Agency Review Draft

Representativeness: Were the data collected from a population sufficiently similar to the 

population of interest and the model-specified population boundaries? Were the sampling and 

analytical methods used to generate the collected data acceptable to this project? Are there 

potentially confounding effects in the data (e.g., season, time of day, location, and scale 

incompatibilities) that need to be addressed so as to not unduly impact the model output?

Comparability: Are there concerns about comparing the data used in modeling to current 

conditions with respect to age or changes in modeling practices? Are there possible concerns that 

the structure of the population from where the data was taken could be different from current 

conditions?

Completeness: Are there any gaps in the data record that could cause concerns with the modeling 

output? To what extent was data interpolation or imputation necessary before the data were 

suitable for modeling purposes?

Sensitivity: Have the data been evaluated in a manner that permits perturbation analysis or 

estimates made as to the sensitivity of the model output to parameter changes? Is the system of 

qualifying or flagging data adequately documented to allow data from different sources to be 

used on the same project (e.g., distinguish actual measurements from estimated values and note 

differences in detection limits)?

The influence and relative importance of the DQIs depends on the purpose and nature of the 

modeling project and further information may be found in Appendix B of this guidance.

When input data are derived from existing models care has to be taken to ensure evaluation of all 

stages of model development: “Some assessments require linking multiple model frameworks 

such that the output from one model is used as input data to another model. For example, air 

quality modeling often links meteorological, emissions, and air chemistry/transport models. 

When employing linked models, the project team should evaluate each component model, as 

well as the full system of integrated models, at each stage of the model development and 

evaluation process” (EPA 2009b).

The importance of knowing the nature and characteristics of the input data is crucial to the 

interpretation of the project’s results.

4.3.2 Data Management Tasks

The QAPP should discuss how data will be managed, how the path of data production in the field 

or laboratory will be traced for the entire project. This includes the following:

 describe or reference the standard record-keeping procedures, and discuss the approach to 

be used for data storage and retrieval of electronic media (see Brilis, et al, 2004);

 discuss the plan for detecting and correcting errors from data conversion (e.g. 

metric/English, units-to-units, and significant figures) as well as for preventing loss of 

data during reduction, reporting, and entry to forms, reports, databases;

 discuss the plan for detecting and correcting errors from data migration to new 

hardware/software (including checks to verify integrity and non-corruption);
QAPP Handbook 84 December 2012 Agency Review Draft

 describe all data-handling equipment and procedures to process, compile, analyze, and 

interpret the model data, including any necessary computer hardware and software; 

 address any specific data-management performance requirements and describe the 

procedures that will be followed to demonstrate acceptability of the necessary

hardware/software configuration; and

 identify who is responsible for each data-management task together with frequency for 

performing these tasks.

In addition to describing the data management tasks, the QAPP should also address how these 

tasks will be documented throughout the project, not already addressed in Section 4.2.8.

4.4 ASSESSMENTS: MODEL EVALUATION ACTIONS (CHECK)

The third part of the Plan-Do-Check-Act paradigm assesses the model by considering the output, 

especially with respect to deviations from what was anticipated. The advice given here is drawn 

from the Standards Clause 7.8, Assessment of Data and Information, and the assessment parts of 

Annex B3.4.4.1, QAPP Requirements for Model Development/Modification, and B3.4.4.2 QAPP 

Requirements for Model Application.

The elements in this section address project checks to see if the model will conform to the 

project’s objectives, and to estimate the effect of any deviations. These activities may be 

performed throughout the project to inform mid-course corrections or at the completion of the 

project.

The roles and responsibilities of project personnel involved in assessing modeling activities 

should be defined in this section. This should include the procedures for corrective actions and to 

whom reports are made. To the extent that the modeling activity includes acquiring data or the

use of existing data, the assessment methods outlined in the previous two chapters are 

appropriate. This section is guidance for the evaluation of the modeling activity.

4.4.1 Model Evaluation Methods and Activities

Model evaluation is defined as the process used to generate information to determine whether a 

model and its analytical results are of a quality sufficient to serve as the basis for a decision. 

Model evaluation is conducted over the life cycle of the project, from development through 

application using the performance criteria established. The effectiveness of evaluation depend 

on good documentation of both quantitative and qualitative comparisons to the performance 

criteria. 

Model evaluation provides information to help answer four main questions (Beck 2000): 

1. How have the principles of documented science been addressed during model 

development? 

2. How is the choice of model supported by the quantity and quality of available data? 

3. How closely does the model approximate the real system of interest? and 
QAPP Handbook 85 December 2012 Agency Review Draft

4. How does the model perform the specified task while meeting the objectives set by QA 

project planning? 

These four factors address two aspects of model quality. The first factor focuses on the intrinsic 

mechanisms and generic properties of a model, regardless of the particular task to which it is 

applied. In contrast, the latter three factors are evaluated in the context of the use of a model 

within a specific set of conditions. Hence, it follows that model quality is an attribute that is 

meaningful only within the context of a specific model application. A model's quality to support 

a decision becomes known when information is available to assess these factors. Evaluation of a 

regulatory model builds upon the four factors described previously and should continue 

throughout the life of the model.

As recommended by the National Research Council (NRC 2007), the QAPP should contain a 

detailed model evaluation plan that includes (but is not limited to): 

 a description of the model and its intended uses; 

 a description of the model’s relationship to data, including the data for inputs, calibration, 

and corroboration;

 a description of how such data and other sources of information will be used to assess the 

ability of the model to meet its intended task; 

 a description of all the evaluation plan elements by using an outline or diagram that 

shows how the elements relate to the model’s life cycle; 

 a description of the factors or events that might trigger the need for major model 

revisions or the circumstances that might prompt users to seek an alternative model (these 

can be fairly broad and qualitative); and 

 identification of the responsibilities, accountabilities, and resources needed to ensure 

implementation of the evaluation plan. 

The goal of model evaluation is to ensure model quality. Documentation of the plan for model 

evaluation provides assurance that the desired model quality will be achieved. 

4.4.2 Evaluation of the Model 

4.4.2.1 Evaluation of Model Development or Modification

Standard Clause B3.4.4.1 states that the evaluation of a model should focus on documented

science, testing strategies and procedures, and documentation of results. The QAPP should 

describe how the project team will ensure that the following issues will be evaluated and 

documented during model development or modification, as appropriate:

 documentation of the science (including peer review of the theory and equations) 

underlying the model and appropriateness of model complexity for the problem;

 testing strategies including individual module tests, integration tests, systems testing, 

acceptance testing, and beta testing, as applicable; data quality and quantity objectives to 

support the model choice;
QAPP Handbook 86 December 2012 Agency Review Draft

 the procedure for each test shall be provided and the process of confirming the test results 

included (i.e., evaluation criteria are to be identified during the initial stages of model 

development); 

 the use of sensitivity and uncertainty analysis with consistency of model structure with all 

relevant inputs described in the conceptual model;

 implementation process for the software and any applicable development standards with 

identification of the model code and code verification, if completed;

 internal quality checks applied during the development process (e.g., design and code 

verification, configuration control procedures, and change control procedures);

 procedures for controlling, documenting, and archiving all significant changes to the 

software and hardware; and

 requirements for project documentation (e.g., design document, source code, 

programmer’s manual, or user guide).

4.4.2.2 Evaluation of Model Application

It is important to note the iterative nature of modeling: “Once a model has been accepted for use

by decision-makers, it is applied to the problem that was identified in the first stages of the 

modeling process. Model application commonly involves a shift from the hindcasting (testing the 

model against past observed conditions) used in the model development and evaluation phases to 

forecasting (predicting a future change) in the application phase. This may involve a 

collaborative effort among modelers and program staff to devise management scenarios that 

represent different regulatory alternatives (EPA 2009b). 

The needs (Standards Clause B3.4.4.2) of a QAPP center on:

 the quality objectives to be used in the model evaluation;

 the requirements for qualitative and quantitative model corroboration;

 use of sensitivity and uncertainty analysis;

 requirements for model post-auditing;

 how the model evaluation will be documented;

 the model parameters and method of estimation; and

 the model calibration methods.

Some model applications may entail trial-and-error model simulations, where model inputs are 

changed iteratively until a desired or stable condition is achieved. When a sufficient quantity of 

data is available, it is common practice to use a portion of their existing data to establish the 

model, and then use the remaining portion to evaluate the performance of the model. The 

rationale on what proportion of data is used to establish the model and how that data is selected 

(randomly or serially across time for example) should be documented. When the model generates 

new data based on the use of existing data, the data fields (variables) and algorithms should be 

traceable back to their origin for this helps prevent the misinterpretation of the project results.

The importance of documentation can be seen in EPA 2009b: “Using a model in a proposed 

decision requires that the model application be transparently incorporated into the public process. 
QAPP Handbook 87 December 2012 Agency Review Draft

This is accomplished by providing written documentation, in the QAPP, of the model’s relevant 

characteristics in a style and format accessible to the interested public, and by sharing specific 

model files and data with external parties, such as technical consultants and university scientists, 

upon request.” 

Finally, it is important to note how the data used to evaluate the model compares to the actual 

scenario the model will be applied to. A model that has good evaluation properties for statistics 

such as averages may be quite poor for predicting extremes. The acceptance criteria established 

should be directly related to the eventual use of the model.

4.4.3 Model Output Sensitivity and Uncertainty Analysis

Sensitivity analysis (Standard Clause B3.4.4.1) is the study of how a model’s response can be 

attributed to known changes in model inputs. Sensitivity analysis is recommended as the 

principal evaluation tool for identifying the most and least important sources of uncertainty in 

environmental models. 

From EPA 2009b: “Uncertainty analysis investigates the lack of knowledge about a certain 

population or the real value of model parameters. Uncertainty can sometimes be reduced through 

further study and by collecting additional data. EPA guidance (e.g., EPA 1997b) distinguishes 

uncertainty analysis from methods used to account for variability in input data and model 

parameters. [V]ariability in model parameters and input data can be better characterized through 

further study but is usually not reducible.” (EPA 2009b)

“Although sensitivity and uncertainty analyses are closely related, sensitivity is algorithmspecific with respect to model “variables” and uncertainty is parameter-specific. Sensitivity

analysis assesses the “sensitivity” of the model to specific parameters and uncertainty analysis 

assesses the “uncertainty” associated with parameter values. Both types of analyses are important 

to understand the degree of confidence a user can place in the model results” (EPA 2009b)”. 

Advice on conducting uncertainty analysis and the establishment of acceptable levels of total 

error (uncertainty) may be found in EPA 2009b, and additional information on uncertainty 

analysis in Good Modelling Practice Handbook (DDPW 1999), and Uncertainty Analysis 

Guidance (EU 2007).

4.5 REVIEW, EVALUATION OF USABILITY: MODELING RESULTS USABILITY 

AND REPORTING REQUIREMENTS (ACT)

The final part of the paradigm concerns the reconciliation of the model outputs to the overall 

project objectives. Peer review is an important part of modeling and must be comprehensively 

addressed. This section offers guidance on Standards Clause 7.9, the evaluation parts of Annex 

B3.4.4.1, QAPP Requirements for Model Development/Modification, B3.4.4.2, QAPP 

Requirements for Model Application, and B3.6, Review, Evaluation of Usability, and Reporting 

Requirements.

The QAPP should state how the results obtained from the project will be reconciled with the 

project objectives and acceptance criteria. This includes the proposed methods for analyzing the 

modeling results and for determining possible anomalies or limitations on the use for the 

intended purpose. 
QAPP Handbook 88 December 2012 Agency Review Draft

The QAPP should describe how departures from assumptions established in the planning phase 

of the modeling process will be assessed, how anomalies will be resolved, and how limitations 

on the use of the data from anomalies and departure from assumptions will be reported to 

decision makers.

4.5.1 Peer Review

For all models, peer review (Standard Clause B3.4.4.2) is an important aspect for acceptance of 

both model development/modification and application as it acts as an independent form of QA to 

ensure the validity of results. “Peer review provides the main mechanism for independent 

evaluation and review of environmental models. Its purpose is two-fold:

 to evaluate whether the assumptions, methods, and conclusions derived from 

environmental models are based on documented scientific principles; and 

 to check the scientific appropriateness of a model for informing a specific regulatory

decision (the latter objective is particularly important for secondary applications of

existing models). 

Information from peer reviews is also helpful for choosing among multiple competing models 

for a specific regulatory application. Finally, peer review is useful to identify the limitations of 

existing models. Peer review is not a mechanism to comment on the regulatory decisions or 

policies that are informed by models (EPA 2006c). Peer review charge questions and 

corresponding records for peer reviewers to answer those questions should be documented 

during the planning stage. For example, peer reviews may focus on whether a model meets the 

objectives or specifications that were established for the project.

All models that inform significant regulatory decisions are candidates for peer review for several 

reasons: 

 model results will be used as a basis for regulatory or policy/guidance decision making; 

 these decisions likely involve significant investment of Agency resources; and 

 these decisions may have inter-Agency or cross-agency implications/applicability.

It is recommended that “a new model should be scientifically peer-reviewed prior to its first 

application; for subsequent applications, the program manager should consider the 

scientific/technical complexity and/or the novelty of the particular circumstances to determine 

whether additional peer review is needed (EPA 1993).” (EPA 2009b)

“Models used for secondary applications (existing EPA models or proprietary models) will 

generally undergo a different type of evaluation than those developed with a specific regulatory 

information need in mind. Specifically, these reviews may deal more with uncertainty about the 

appropriate application of a model to a specific set of conditions than with the science underlying 

the model framework.” (EPA 2009b)
QAPP Handbook 89 December 2012 Agency Review Draft

The following aspects of a model should be peer-reviewed to establish scientific credibility (EPA 

1993):

 appropriateness of input data; 

 appropriateness of boundary condition specifications; 

 documentation of inputs and assumptions; 

 applicability and appropriateness of selected parameter values;

 evaluation of model uncertainty;

 documentation and justification for adjusting model inputs to improve model 

performance (calibration); 

 model application with respect to the range of its validity; and 

 supporting empirical data that strengthen or contradict the conclusions that are based on 

model results. 

To be most effective and maximize its value, external peer review should begin as early in the 

model development phase as possible. The importance of using independent model review is 

extremely important and the decision whether to utilize the Agency’s Peer Review system made 

as soon as possible (EPA 2006c).

4.5.2 Description of Model Documentation

The importance of documentation can be seen from EPA 2009b: “Documentation enables 

decision makers and other model users to understand the process by which a model was 

developed and used. During model development and use, many choices are made and options 

selected that may bias the model results. Documenting this process and its limitations and 

uncertainties is essential to increase the utility and acceptability of the model outcomes. 

Modelers and project teams should document all relevant information about the model to the 

extent practicable, particularly when a controversial decision is involved. In legal proceedings, 

the quality and thoroughness of the model’s written documentation and the Agency’s responses 

to peer review and public comments on the model can affect the outcome of the legal challenge.” 

The QAPP should describe how the modeling process, the model framework, the model, the 

sensitivity and uncertainty analyses, and the model outputs will be documented. The 

documentation should include a clear explanation of the model’s relationship to the scenario of 

the particular application. This explanation should describe the limitations of the available 

information when applied to other scenarios. The evaluation, verification, and validation 

performed on the model should be documented. This documentation should include a clear 

explanation of the model’s relationship to the final goal of the project or task. Disclosure about 

the state of science used in a model and future plans to update the model can help establish a 

record of reasoned, evidence-based application to inform decisions. 

It is also advisable to include in the QAPP any particular memory requirements, necessity for 

specific software for model application, and any issues with model portability.
QAPP Handbook 90 December 2012 Agency Review Draft

4.5.3 Specifications for Model Maintenance and User Support 

The project team should determine and document in the QAPP, during the planning stage, the 

expected maintenance and user support needed throughout the life cycle of the model.

4.5.4 Reports to Management

The QAPP should identify the frequency and distribution of reports issued to inform 

management (EPA or otherwise) of the project status. QAPP authors may choose to identify the 

preparer and the recipients of the reports, and any specific actions recipients are expected to take 

as a result of the reports. 

Periodic QA management reports are intended to ensure that managers and stakeholders are kept 

informed of project status and results of all QA assessments. Efficient communication of project 

status and problems will allow project managers to implement timely and effective corrective 

actions so that data produced for the project can meet the DQOs.

The QAPP should describe the content of each QA management report that will be generated for 

the project, including an evaluation of measurement error as determined from the assessments. 

Assessment checklists, reports, requests for corrective action letters, and the corrective response

letters should be included as attachments or referenced in the QA management reports.

The following issues may be included in QA management reports:

 Graphs, charts, diagrams helping interpreting results;

 conformance of project activities to QAPP requirements and procedures;

 status of project and schedule delays;

 deviations from the approved QAPP and approved amendments to the QAPP;

 results of interim model evaluation activities and corrective actions;

 model sensitivity and uncertainty analyses; and

 limitations on the use of model output.

Although these issues listed may be addressed in QA management reports, they should also be 

included in the QA/QC section of the final project report. The final project report should, at a 

minimum, give a reconciliation of project data with project objectives, data summary (including 

tables, charts, and graphs), a summary of major problems encountered and their resolution, and 

any additional data quality concerns together with conclusions and recommendations.
QAPP Handbook 91 December 2012 Agency Review Draft

GLOSSARY

Several definitions given in CIO Policy 2106.0, CIO Procedure 2106-P-01.0, CIO Standard 

2106-S-01, and CIO Standard 2106-S-02 are repeated below for added clarity and utility to this 

Guidance. 

Approved: The documented determination that the proposed quality document is suitable for the 

intended purpose and meets the requirements is specified in the applicable Quality Standard.

Assessment: The evaluation process used to measure the performance or effectiveness of a 

system and its elements. As used here, assessment is an all-inclusive term used to denote any of 

the following: audit, performance evaluation, management review, peer review, inspection, or 

surveillance.

Conceptual Model: This describes the scientific and engineering process under investigation. It 

is used as a tool for organizing information about the current state of knowledge and 

understanding of the project, as well as for documenting key theoretical and practical 

assumptions underlying the data and information collection.

Corrective Action: An action to eliminate the cause of a nonconformity or undesirable situation 

and to prevent recurrence.

Data Standard: A documented consensus-based agreement on the format and definition of data.

Data Usability: The process of determining and ensuring that the quality of the data produced 

meets the intended use of the data.

Document: Recorded information regardless of physical form or characteristics including 

individual records or items of non-record materials.

Environmental Data: Any data or information pertaining to the environment that describe 

measured outputs from processes; environmental conditions in a specific location; ecological 

effects and consequences; health effects and consequences; biological, chemical, and 

radiological conditions; or the performance of environmental technology. Environmental data 

include information collected directly from measurements, produced from models, and compiled 

from other sources such as databases, information systems, literature, or the internet.

Environmental Data Collection: The process of acquiring or gathering environmental data 

through various means including, but not limited to, sampling and analysis activities, retrieval 

from information systems and literature sources, and receipt from EPA partners and the regulated 

community.

Environmental Data Operations: The work performed to collect, produce, or use 

environmental data.
QAPP Handbook 92 December 2012 Agency Review Draft

Environmental Data Production: The process of generating environmental data through 

various means including, but not limited to, the use of measurement instrumentation, information 

technology, computer models, and data analysis tools (e.g., statistics, risk assessment methods).

Equivalent Document: A set of documents that contains all the information and management 

controls (signatures) as the required documents used in the Standard.

Extramural Agreement: Legal agreement between EPA and a non-EPA organization for the 

acquisition of items or services by EPA or financial assistance to a non-EPA organization. Such 

agreements include acquisition agreements (e.g., contracts, work assignments, delivery orders, 

task orders), assistance agreements (e.g., cooperative agreements, research grants, state and local 

grants), and EPA-funded IAs with other governmental entities.

Graded Approach: The process of basing the level of application of managerial controls 

applied to an item or work according to the intended use of the results and the degree of 

confidence needed in the quality of the results.

Guidance: A non-mandatory compilation of advice, examples, best practices, or past experience. 

Guidance may supplement procedures.

Handbook: A non-mandatory compilation of advice, examples, best practices, or past 

experiences, may be revised according to the issuing Office’s Peer Review Policy.

Information: For purposes of this Standard, information means any communication or 

representation of knowledge such as facts or data, in any medium or form, including, but not 

limited to, textual, numerical, graphic, cartographic, narrative, or audiovisual forms.

Information System: An organized collection, storage, and presentation system of data for 

decision making, progress reporting, and for planning and evaluation of programs. It can be 

either manual or computerized, or a combination of both.

Information Technology: The study, design, development, implementation, support, or 

management of computer-based information systems, particularly software applications and 

computer hardware.

Informative: Non-mandatory advice on how to achieve a goal or meet a requirement.

Integrity (Information): The assurance that the information is protected from unauthorized 

access or change and is not compromised through corruption or falsification.

Life Cycle: The life span of a product or service from its initial planning and development, to its 

use and maintenance, and to its final closure or disposal.

Management Controls: A system of management functions to enable managers to determine 

that the operations of a program or organization satisfy predetermined goals and objectives, that

performance is in line with standards and specifications, and to implement any remedial actions 
QAPP Handbook 93 December 2012 Agency Review Draft

needed to ensure that human and other resources are being used in the most effective and 

efficient way possible in achieving the organization’s mission.

Management System: A system to establish policy and objectives and to achieve those 

objectives (ISO 9001). A management system may describe the policies, objectives, principles, 

authority, responsibilities, accountability, and implementation plan of an organization for 

conducting work and producing products and services. Management systems include ISO 9001 

on quality management, ISO 14001 on environmental management, and OHSAS 18000 on 

occupational health and safety.

Metadata: The information about data required to facilitate its use, understanding, and 

management. For example, metadata should answer questions about data such as why they were 

collected, how they were collected and by whom, what was done to the data, what they were 

used for, what were their limitations, and what were the acceptance criteria.

Model: A simplification of reality that is constructed to gain insights into select attributes of a 

physical, biological, economic, or social system. A formal representation of the behavior of 

system processes, often in mathematical or statistical terms. The basis can also be physical or 

conceptual.

Normative: A mandatory set of rules that must be followed.

Organization: A company, corporation, firm, enterprise, or institution, or part thereof, whether 

incorporated or not, public or private, that has its own functions and administration. In the 

context of this Standard, an EPA organization may be an Office, Region, National Research 

Center or Laboratory, or a subunit such as a division, branch, section, or team.

Objectivity (Information): The assurance that information is presented in an accurate, clear, 

complete, and unbiased manner, and, as a matter of substance, is accurate, reliable, and unbiased.

Peer Review: A review conducted by qualified individuals or organizations who are independent 

of those who performed the work, but are collectively equivalent in technical expertise to those 

who performed the original work.

Policy: A high-level statement about an Agency requirement designed to influence and 

determine decisions, actions, and other matters. It is usually driven by statute, executive order, 

the mandate of an oversight agency or Congress, or the head of the organization.

Practice: 

 an expected procedure or way of doing of something.

 the carrying out or exercise of an expected procedure or way of doing of something.

Procedure: The required steps, course of action, or processes needed to accomplish or satisfy a 

policy.
QAPP Handbook 94 December 2012 Agency Review Draft

Process: A set of interrelated resources and activities which transforms inputs into outputs. 

Examples of processes include analysis, design, data collection, operation, fabrication, and 

calculation.

Product: The intended result or final output of an activity or process that is disseminated or 

distributed among EPA organizations or outside of EPA.

Quality: The totality of features and characteristics of a product or service that bear on its ability

to meet the stated or implied needs and expectations of the user.

Quality Assurance (QA): A management or oversight function that deals with setting policy

and running an administrative system of management controls that cover planning, 

implementation, review and maintenance to ensure products and services are meeting their 

intended use.

Quality Assurance Manager (QAM): The individual designated as the principal manager 

within the organization having management oversight and responsibilities for planning, 

documenting, coordinating, and assessing the effectiveness of the QMS for the organization. 

Note: Other personnel having QA or QC duties may be referred to as QA Officer and QA 

Coordinator.

Quality Assurance Project Plan (QAPP): A document describing in comprehensive detail the 

necessary QA, QC, and other technical activities that must be implemented to ensure that the

results of the work performed will satisfy the stated performance objectives and criteria.

Quality Control (QC): The overall system of technical activities that measure the attributes and 

performance of a process, item, or service against defined standards to verify that they meet the 

stated requirements established by the customer; operational techniques and activities that are 

used to fulfill requirements for quality.

Quality Management: That aspect of the overall quality management system of the 

organization that determines and implements the quality policy. Quality management typically 

includes strategic planning, allocation of resources, and other systematic activities (e.g., 

planning, implementation, and assessment) pertaining to the application of quality practices to 

the organization’s programs.

Quality Management Plan (QMP): A formal document or manual that describes the QMS in 

terms of the organizational structure, functional responsibilities of management and staff, lines of 

authority, and required interfaces for those planning, implementing, and assessing all activities 

conducted.

Quality Management System (QMS): A structured and documented management system 

describing the policies, objectives, principles, organizational authority, responsibilities, 

accountability, and implementation plan of an organization for ensuring quality in its work 

processes, products (items), and services. The QMS provides the framework for planning, 
QAPP Handbook 95 December 2012 Agency Review Draft

implementing, documenting, and assessing work performed by the organization and for carrying 

out required QA and QC activities.

Quality Program: The totality of management controls, processes, and documentation in 

planning, implementation, and assessment of applying quality to the creation of Agency products 

and services.

Record: A document stating results retrieved or providing evidence of activities (ISO 

9000:2005). NOTE: A federal record is an information resource in any format that is needed to 

describe Agency activities (44 U.S.C. § 3301).

Required Element: An element whose presence is obligatory in a Standard.

Requirement: An expression of the content of a Standard conveying a criterion to be fulfilled if 

compliance is to be claimed and from which no deviation is permitted.

Service: A discrete function that performs operations and returns a set of results to an external 

requester.

Specification: A document stating requirements and which refers to or includes drawings or 

other relevant documents. Specifications should indicate the means and the criteria for 

determining conformance.

Standard: An accepted, consensus-based specification which defines systems, processes, 

methodologies, or practices. It provides a basis for assuring consistent and acceptable minimum 

levels of quality, performance, safety, and reliability. Standards usually are included in or 

accompany procedures.

Standard Operating Procedure (SOP): A written document that details the method for an 

operation, analysis, or action with thoroughly prescribed techniques and steps, and that is 

officially approved as the method for performing certain routine or repetitive tasks.

Usability Assessment: The evaluation of data based upon the results of data validation and 

verification for the decision(s) being made. Reviewers assess whether the process execution and 

resulting data meet quality objectives based on the criteria given in the QAPP.

User: An organization, group, or individual that utilizes the results from environmental 

programs, or the customer for whom the results or products were collected or created.

Utility (Information): The assurance that information is useful for its intended purpose.

Validation (Information): The confirmation by examination and provision of objective 

evidence that the particular requirement for which the information is intended are fulfilled; the 

process of determining whether the specifications were appropriate and that the verified results 

will meet the data user’s needs.
QAPP Handbook 96 December 2012 Agency Review Draft

Verification (Information): The confirmation by examination and provision of objective 

evidence that validated information fulfills specified requirements; the process of checking 

whether the information met the project’s specifications as part of an agreement or contract.
QAPP Handbook 97 December 2012 Agency Review Draft

REFERENCES

40 CFR 3, Code of Federal Regulations (annual edition). Title 40: Protection of Environment. 

Part 3: Special rules governing certain information (from regulatory programs). Washington, 

D.C.

http://ecfr.gpoaccess.gov/

40 CFR 51 Appendix W, Code of Federal Regulations (annual edition). Title 40: Protection of 

Environment. Part 51: Requirements for Preparation, Adoption, and Submittal of

Implementation Plans. Washington, D.C. 

http://ecfr.gpoaccess.gov/

40 CFR 403.12(b), Code of Federal Regulations (annual edition). Title 40: Protection of 

Environment. Subchapter N: Effluent Guidelines and Standards. Part 403: General Pretreatment 

Regulations for Existing and New Sources of Pollution. Washington, D.C. 

http://ecfr.gpoaccess.gov/

40 CFR 763, Code of Federal Regulations (annual edition). Title 40: Protection of Environment.

Part 763, Subpart E: Asbestos-Containing Material in Schools. Washington, D.C. 

http://ecfr.gpoaccess.gov/

49 CFR 171, Code of Federal Regulations (annual edition). Title 49: Transportation. Part 171: 

Definition of Hazardous Material. Washington, D.C.

http://ecfr.gpoaccess.gov/

ANSI 2004, American National Standards Institute and American Society for Quality, Quality 

Systems for Environmental Data and Technology Programs: Requirements with Guidance for 

Use (ANSI/ASQ E4-2004).

http://asq.org/quality-press/display

ASTM 1999, ASTM Standard D4840-99, 1999, Standard Guide for Sampling Chain-of-Custody

Procedures, ASTM International, West Conshohocken, PA.

http://astm.org

Beck, B, 2002 Model Evaluation and Performance. In: Encyclopedia of Environmetrics. A.H. ElShaarawi and W.W. Piegorsch, eds., John Wiley & Son, New York.

Box, G. E. P., W. G. Hunter, and J. S. Hunter, 1978. Statistics for Experimenters. John Wiley.

New York.

Brilis, G. M, J. C. Worthington, A. D. Wait, 2000. Quality Science in the Courtroom: U.S. 

EPA Data Quality and Peer Review Policies and Procedures compared to the Daubert Factors,

Environmental Forensics, vol. 1 p197-203.

Brilis, G. M., J. Lyon, J.C. Worthington, and R. Lysakowski, 2004. Auditing Electronic 

Record-keeping Systems used in Scientific Research and Development, Quality Assurance, vol. 

11 p5-24.

Cochran, W. G., and D. R. Cox, 1957. Experimental Designs. John Wiley. New York.
QAPP Handbook 98 December 2012 Agency Review Draft

Currie, L. A., 1995. Nomenclature in Evaluation of Analytical Methods Including Detection and 

Quantification Capabilities. Pure and Applied Chemistry, Vol. 67, No. 10, pp. 1699–1723.

DDPW 1999, Good Modelling Practice Handbook, Dutch Department of Public Works.

http://harmoniqua.wau.nl/public/Reports/Existing%20Guidelines/GMP111.pdf

EPA 1997a, Procedures for Preparing Emission Factor Documents (EPA-454/R-95-015)

U.S. Environmental Protection Agency.

http://www.epa.gov/ttnchie1/efdocs/procedur.pdf

EPA 1997b, Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM).

NUREG-1575, EPA/402/R-97/016, U.S. Environmental Protection Agency.

www.marssim.com

EPA 1999, Contract Laboratory Program, National Functional Guidelines for Organic Data 

Review. U.S. Environmental Protection Agency.

www.epa.gov/superfund/programs/clp/guidance.htm

EPA 2000, Guidance on Technical Audits and Related Assessments for Environmental Data 

Operations, EPA QA/G-7, January 2000, U.S. Environmental Protection Agency. 

http://www.epa.gov/quality

EPA 2001a, Requirements for Quality Assurance Project Plans, EPA QA/R-5. EPA/240/B01/003, U.S. Environmental Protection Agency.

http://www.epa.gov/quality

EPA 2001b, ORD Policies and Procedures Manual, Chapter 13 Quality Assurance, U.S.

Environmental Protection Agency.

http://dcordhqapps1.eoa.gov/orma/policies.nsf/webpolicy?OpenView

EPA 2002a, Guidance for Quality Assurance Project Plans, EPA QA/G-5. EPA/240/R-02/009,

U.S. Environmental Protection Agency.

http://www.epa.gov/quality

EPA 2002b, Guidance for Choosing a Sampling Design for Environmental Data Collection, 

EPA QA/G-5S, U.S. Environmental Protection Agency.

http://www.epa.gov/quality

EPA 2002c, Guidance for Quality Assurance Project Plans for Modeling, EPA QA/G-5M U.S. 

Environmental Protection Agency.

http://www.epa.gov/quality

EPA 2005a, Uniform Federal Policy for Quality Assurance Project Plans, Intergovernmental 

Data Quality Task Force (EPA-505-B-04-900A) U.S. Environmental Protection Agency. 

http://www.epa.gov/fedfac/documents/qualityassurance.htm

EPA 2005b, Workbook for Uniform Federal Policy for Quality Assurance Project Plans,

Intergovernmental Data Quality Task Force (EPA-505-B-04-900C), U.S. Environmental 

Protection Agency.

http://www.epa.gov/fedfac/documents/qualityassurance.htm
QAPP Handbook 99 December 2012 Agency Review Draft

EPA 2005c, OSWER Directive 9272.0-17 Implementation of the Uniform Federal Policy for 

Quality Assurance Project Plans (UFP-QAPP) at Federal Facility Hazardous Waste Sites, June 

7, 2005, U.S. Environmental Protection Agency.

www.epa.gov/fedfac/pdf/oswer_qapp_directive.pdf

EPA 2005d, OSWER Directive 9272.0-20 Applicability of the Uniform Federal Policy for 

Quality Assurance Project Plans (EPA 505-04-900A, December 21, 2005, U.S. Environmental 

Protection Agency.

www.epa.gov/fedfac/pdf/oswer_qapp_directive.pdf

EPA 2006a, Guidance on Systematic Planning Using the Data Quality Objectives Process, EPA 

QA/G-4, U.S. Environmental Protection Agency. 

http://www.epa.gov/quality

EPA 2006b, Data Quality Assessment: Statistical Methods for Practitioners, EPA QA/G-9S, 

U.S. Environmental Protection Agency. 

http://www.epa.gov/quality

EPA 2006c, Peer Review Handbook, U.S. Environmental Protection Agency. 

http://www.epa.gov/peerreview/pdfs/prhandbk.pdf

EPA 2006d, Pre-Dissemination Review Guidelines, U.S. Environmental Protection Agency.

http://intranet.epa.gov/quality/informationguidelines/pdf/pdr-guidelines.pdf

EPA 2008a, Quality Policy, CIO Policy 2106.0, Oct. 20, 2008 (“Quality Policy”), U.S. 

Environmental Protection Agency. 

http://www.epa.gov/irmpoli8/policies/21060.pdf

EPA 2008b, Procedure for Quality Policy, CIO 2106-P-01.0, Oct. 20, 2008 (“Quality 

Procedure”), U.S. Environmental Protection Agency.

http://www.epa.gov/irmpoli8/policies/2106p01.pdf

EPA 2008c, Guidance on Environmental Data Verification and Data Validation, EPA QA/G-8,

U.S. Environmental Protection Agency.

http://www.epa.gov/quality

EPA 2009a, EPA Records Management Policy, U.S. Environmental Protection Agency.

http://epa.gov/records

EPA 2009b, Guidance on the Development, Evaluation, and Application of Environmental 

Models, Council for Regulatory Environmental Modeling (EPA/100/K-09), U.S. Environmental 

Protection Agency.

http://www.epa.gov/crem/library/cred_guidance_0309.pdf.

EPA 2009c, Guidance for Labeling Externally Validated Laboratory Analytical Data for 

Superfund Use, OSWER No. 9200.1-85 (EPA 540-R-08-005), U.S. Environmental Protection 

Agency.

www.epa.gov/superfund/policy/pdfs/EPA-540-R-08-005.pdf
QAPP Handbook 100 December 2012 Agency Review Draft

EPA 2010, Technical Basis for the Lowest Concentration Minimum Reporting Level (LCMRL) 

Calculator, Office of Water (EPA 815-R-11-001), U.S. Environmental Protection Agency.

http://water.epa.gov/drink

EPA 2012, EPA Regional Guidance for the Preparation of Quality Assurance Program Plans

(R9/3.1), U.S. Environmental Protection Agency. 

http://www.epa.gov/region09/qa/pdfs/mngmt-plan_guidance_2012.pdf

EPA 2013a, Quality Standard for Environmental Data Collection, Production, and Use by EPA 

Organizations, CIO 2106-S-01.0, In Preparation (“Internal Standard”), U.S. Environmental 

Protection Agency,

http://www.epa.gov/quality

EPA 2013b, Quality Standard for Environmental Data Collection, Production, and Use by NonEPA (External) Organizations, CIO 2106-S-02.0, In Preparation (“External Standard”), U.S. 

Environmental Protection Agency. 

http://www.epa.gov/quality

EPA 2013c, Guidance on Quality Management Plans, CIO Guidance QMP Handbook, In 

Preparation (“QMP Guidance”), U.S. Environmental Protection Agency.

http://www.epa.gov/quality

EU 2007, Uncertainty Analysis Guidance, European Commission 2007

http://www.harmoni-ca.info/toolbox/Model_Uncertainty/index.php

Hubaux, A. and G. Vos, 1970, Decision and Detection Limits for Linear Calibration Curves,

Analytical Chemistry. Vol. 42, No. 8, pp. 849-855.

IQG 2001. Information Quality Act Section 515(a) of the Treasury and General Government 

Appropriations Act for Fiscal Year 2001 (Public Law 106-554; H.R. 5658).

www.nrc.gov/public-involve/info-quality/pl106-554.pdf

NRC 2007, Models in Environmental Decision Making, Committee on Models in the Regulatory 

Decision Process, Board on Environmental Studies and Toxicology, Division on Earth and Life 

Studies. National Academic Press, Washington D.C.

Manno, J., R.Smardon, J.V. DePinto, E.T. Cloyd, and S. Del Grando 2008. The Use of 

Models in Great Lakes Decision Making: An Interdisciplinary Synthesis. Randolph G. Pack 

Environmental Institute, College of Environmental Science and Forestry. Occasional Paper 16.

Skoog, D. A. 1985, Principles of Instrumental Analysis, third edition. Saunders College 

Publishing. Philadelphia, Pennsylvania.

Taylor, J. K. 1987, Quality Assurance of Chemical Measurements. Lewis Publishers. Chelsea, 

Michigan.

Ziliak, S.T. 2011, Matrixx v. Siracusano and Student v. Fisher, Significance, vol 8, p 131-134.
QAPP Handbook A-1 Agency Review Draft 

APPENDIX A

CHECKLIST OF QAPP ELEMENTS 

QAPPs should, at a minimum, address all elements detailed in this Handbook as it supports the 

Internal and External Standards. In some cases, certain elements will not be appropriate for a 

particular project. Elements that do not apply can be addressed with a simple statement of why

the information is not relevant or with a cross reference to another approved document in which 

the information appears.

This checklist shows the section number and title for each element with columns for:

 describing why the element is important;

 indicating what might suffice for the element in a simple project;

 listing what additional information might be necessary for the element in a more complex 

project;

 noting whether the element was discussed in the QAPP;

 noting where in the QAPP it was discussed; and 

 explaining why an element was not included.

This checklist is meant to be useful either as a tool for the developer of the QAPP to ensure all

elements are included, or as a tool for reviewers who are checking to be sure that all elements are 

appropriately included. 

This checklist is organized by the four primary steps in the project life cycle: Plan-Do-Check-Act

and presented with steps 1 and 3 shaded, steps 2 and 4 un-shaded for clarity. The elements for 

Chapter 2 are presented first, then Chapter 3, and finally Chapter 4. To facilitate use, the 

generation of new data (Chapter 2) is shown in black, the use of existing data (Chapter 3) in 

green, and the application to models (Chapter 4) in blue. 

In order to help discuss the degree to which the QAAP elements need to be addressed, a simple 

case is contrasted to a more complex case; an example of the graded approach being used. These 

two cases are, by nature of being examples, fairly simplistic as the detail of documentation 

depends on the importance of the project, an exploratory study needing significantly less 

documentation than one involving a proposed regulation.
QAPP Handbook A-2 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

2.2.1 Title, Version, 

and Approval/

Sign-off

To meet the requirements of 

Clause 7.5.1 of the 

Standards for an approved 

QAPP prior to initiating 

work

Single page with title and 

required signatures

Revisions history tracking 

all changes made to the 

QAPP over time

2.2.2 Document 

Format and 

Table of 

Contents

To make it easy for the 

project team and reviewers 

to find the information they 

need

Table of contents and page 

numbers

Document control format

2.2.3 Distribution 

List

To be sure that everyone 

who needs it has access to, 

and awareness of, the latest 

version of the QAPP

Included on the title page or 

in the project organization 

section

A separate section with 

contact information and 

indication of under what 

conditions each individual 

needs to be made aware of 

revisions to the QAPP

2.2.4 Project 

Organization 

and Schedule

To ensure all key project 

personnel are aware of their 

responsibilities and the 

timeframe for completion

Names of key project 

personnel and period of 

performance for project

Organizational chart of all 

key project roles with names 

and Gantt or similar tracking 

chart of schedule

2.2.5 Project 

Background, 

Overview, and 

Intended Use of 

Data

To provide sufficient 

information as a foundation 

for the project goals and to 

clarify the expected uses of 

the data

General overview and 

statement of how data and 

models will be used in this 

project

Detailed project history, 

references to other 

background documentation, 

regulatory basis for data use

2.2.6 Data/Project 

Quality 

Objectives and 

Measurement 

Performance 

Criteria

To ensure that the data or 

models collected or used in 

the project are of sufficient 

quality to support project 

decisions

Project goals, data required 

to meet those goals, 

statement of what criteria 

the data must meet to be 

acceptable for project use

7-step DQOs, MQOs for all 

DQIs, statement of tolerable 

error ranges for project 

decisions

2.2.7 Special 

Training 

Requirements 

and 

Certification

To be sure that the project 

has qualified personnel to 

perform all necessary 

functions

List of specialized training 

required in the project

Matrix of special training 

needs, personnel who are 

qualified, and timeframe of 

qualifications

2.2.8 Documentation 

and Records 

Requirements

To document project 

process, outcomes, and 

supporting quality 

information, to meet clause 

7.10 of the Standards

Statement declaring where 

project documentation will 

be stored, including, but not 

limited to: QAPP, data 

packages, assessment 

records, interim and final 

reports

Detailed documentation and 

storage requirements per 

regulation or contractual 

obligation. These may be 

indicated in the project 

schedule as well.
QAPP Handbook A-3 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

2.3.1 Sample 

Collection 

Procedure, 

Experimental 

Design, and 

Sampling Tasks

To define the way the data 

will be collected and state 

whether it is a probabilitybased or judgmental design

Define population of 

interest, sample design and 

rationale for that design, 

number/time/location and 

types of samples to be 

collected 

As appropriate, discuss 

multistage sampling, 

randomization procedures, 

stratification rationale, or 

any details that support the 

selection of samples

2.3.2 Sampling 

Procedures and 

Requirements

To ensure that appropriate 

sampling methods are 

selected to meet project 

needs and produce data of 

sufficient quality for project 

decision making

Include or reference SOPs 

for sample collection, onsite preservation, and 

cleaning and 

decontamination of 

sampling equipment

Include media-specific 

sample collection 

procedures, safety issues 

involved in sample 

collection, sample container 

descriptions, etc.

2.3.3 Sample Custody 

Procedures, and 

Documentation

To ensure sample 

authenticity and to avoid 

sample loss

Describe sample 

documentation and handling 

procedures and include a 

sample chain-of-custody

form

List US DOT regulations 

and how they will be met, 

explain sample numbering 

schema, define procedures 

for introducing PE samples

2.3.4 Analytical 

Methods 

Requirements

and task 

Descriptions

To be sure appropriate 

measurement methods exist 

to achieve project quality 

objectives

Describe measurement 

techniques, such as 

counting, visual 

discrimination, or analytical 

methods

Details of analytes to be 

measured, analytical 

method, SOPs, data-package 

requirements, analytical 

laboratory contact 

information, and MQOs

2.3.5 Quality Control 

Requirements

To provide confidence that 

the project data are of 

suitable quality to be used 

for their intended purpose

Number and types of QC 

samples to address sources 

of measurement error

Table of sources of potential 

measurement error and the 

corresponding QC samples 

that will be used to address 

those uncertainties 

2.3.6 Instrument/

Equipment 

Testing 

Calibration and 

Maintenance 

Requirements, 

Supplies and 

Consumables

To avoid poor instrument 

performance that could 

impact project data quality

Document roles and 

responsibilities and 

procedures for sample 

collection and measurement 

instrument inspection, 

calibration, and maintenance

Include instrument 

measurement uncertainty, 

traceability of calibration 

equipment, and projectspecific schedule for 

inspection, calibration, and 

maintenance

2.3.7 Data 

Management 

Requirements

To ensure data integrity Describe data handling from 

generation, to use, to final 

storage; include copies of 

data entry forms, reports and 

description of databases; 

specify any special 

requirements for data such 

as CBI or hardware/software 

Include SOPs for data 

management, specify project 

personnel’s roles and 

responsibilities regarding 

data management, describe 

how metadata will be 

gathered and stored with 

project data
QAPP Handbook A-4 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

2.4.1 Technical 

Systems 

Assessments

To determine if the field 

sampling team, laboratory, 

or analysts have sufficient 

technical capabilities to 

generate data of appropriate 

quality for project use

If project-specific needs are 

minimal, noting 

accreditation to an 

appropriate technical system 

standard may be sufficient ; 

note that if any projectspecific needs are more 

stringent than the standards, 

then project-specific 

assessments should be 

conducted

Describe plans for 

assessments of field 

sampling, on-site and offsite laboratory analyses

2.4.2 Performance 

Audits of

Measurement 

and Analytical 

Systems

To directly test that the 

measurement performance is 

adequate for project 

purposes

Document plans and 

acceptance criteria for split 

samples and PE samples, if 

appropriate

Specify schedule, analyte 

selection, traceability, 

spiking levels, and sample 

types (matrix, direct, singleblind, or double-blind), and 

include the sources and 

estimated costs for these 

samples 

2.4.3 Surveillance of 

Operations

To verify that project 

activities are conducted as 

planned

State when surveillance will 

occur (under what 

conditions or by set 

timeframe), how it will be 

conducted, how feedback 

will be provided and 

incorporated, and if 

surveillance leads to a 

temporary or permanent 

work stoppage, address how

that will be handled

Provide details of the 

triggering events for 

surveillance assessments, an 

SOP for their conduct, a list 

of who will be notified of 

any non-conformances 

observed during the 

assessments, and how the 

surveillance assessments 

will be documented

2.4.4 Audits of Data 

Quality 

To be sure that QC data are 

used to support data quality, 

and to determine if the data 

are replicable

Define the schedule (based 

on timeframe or triggering 

events) and scope for audits 

of data quality 

Provide an SOP for 

conducting audits of data 

quality 

2.4.5 Interim 

Assessments of 

Data Quality

To be sure data collection is 

proceeding according to 

plan

Include statement 

encouraging project team 

members to alert 

management if they sense 

anything isn’t going quite 

right

Indicate points during the 

project timeline at which 

interim assessments of data 

quality will be conducted; 

state who will be 

responsible for their conduct 

and how they will be 

documented
QAPP Handbook A-5 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

2.4.6 Qualitative and 

Quantitative 

Comparisons to 

Acceptance 

Criteria

To determine if the project 

quality objectives are met

State how comparisons to 

qualitative and quantitative 

MQOs will be evaluated; 

describe other criteria (e.g., 

publication in a peerreviewed journal) that might 

be important for the project

Set specific comparison 

methodology for each DQI, 

such that there will be a 

definitive “pass/fail” for 

each; describe contingencies 

in case of “fail”

2.4.7 Evaluation of 

Unconventional 

Measurements

To be sure that any unusual 

laboratory methods are fully 

documented and understood

If no unconventional 

measurement methods will 

be/were used, just state that 

here

Describe the reason for 

using unconventional 

measurement methods, 

provide SOPs for their 

implementation and how

efficacy will be assessed

2.4.8 Evaluation of 

Unconventional 

Monitoring 

Projects

To be sure that any unusual 

sample collection methods 

are fully documented and 

understood

If no unconventional sample 

collection methods will 

be/were used, just state that 

here

Describe the reason for 

using unconventional 

sample collection methods, 

provide SOPs for their 

implementation and how

efficacy will be assessed

2.5.1 Data 

Verification and 

Validation 

Targets and 

Methods

To determine if data have 

met the project quality 

objectives

Provide a standard data 

verification and validation 

method or procedure that 

has been reviewed to ensure 

it meets project needs

Develop project-specific 

verification and validation 

schema that incorporate QC 

data and metadata

2.5.2 Quantitative 

and Qualitative 

Evaluations of 

Usability

To ensure that project 

decisions are supported by 

data of sufficient quality for 

project needs

State who will be part of the 

evaluation of data usability, 

how it will be conducted 

and documented

Define contingencies for 

any issues that may be 

identified during the 

evaluation of data usability

2.5.3 Potential 

Limitations on 

Data 

Interpretation

To ensure that the data are 

not stretched beyond their 

appropriate use

Describe what actions will 

be taken if project data are 

deemed unusable for their 

intended project purpose

Describe how any 

limitations will be 

documented and stored in 

the project metadata 

2.5.4 Reconciliation 

with Project 

Requirements

To determine if project 

requirements have been met 

Clearly state how the data 

verification, validation, and 

usability results will be used 

to determine if project 

requirements have been met; 

describe how the five steps 

of the DQA process will be 

conducted

State specific exceptions to 

statistical signficance that 

will be overturned in favor 

of “practical signficance” 

should they occur; define 

the steps to be taken for 

contingencies if data do not 

support requirements

2.5.5 Reports to 

Management

To document project 

outcomes

Define schedule and content 

for reports to management

Provide report templates
QAPP Handbook A-6 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

3.2.1 Title, Version, 

and Approval/ 

Sign-off

To meet the requirements of 

Clause 7.5.1 of the 

Standards for an approved 

QAPP prior to initiating 

work

Single page with title and 

required signatures

Revisions history tracking 

all changes made to the 

QAPP over time

3.2.2 Document 

Format and 

Table of 

Contents

To make it easy for the 

project team and reviewers 

to find the information they 

need

Table of contents and page 

numbers

Document control format

3.2.3 Distribution 

List

To be sure that everyone 

who needs it has access to, 

and awareness of, the latest 

version of the QAPP

Included on the title page or 

in the project organization 

section

A separate section with 

contact information and 

indication of under what 

conditions each individual 

needs to be made aware of 

revisions to the QAPP

3.2.4 Project 

Organization 

and Schedule

To ensure all key project 

personnel are aware of their 

responsibilities and the 

timeframe for completion

Names of key project 

personnel and period of 

performance for project

Organizational chart of all 

key project roles with names 

and Gantt or similar tracking 

chart of schedule

3.2.5 Project 

Background, 

Overview, and 

Intended Use of 

Data

To provide sufficient 

information as a foundation 

for project goals and to 

clarify expected uses of the 

data

General overview and 

statement of how data and 

models will be used in this 

project

Detailed project history, 

references to other 

background documentation, 

regulatory basis for data use

3.2.6 Data/Project 

Quality 

Objectives and 

Measurement 

Performance 

Criteria

To ensure that the data or 

models collected or used in 

the project are of sufficient 

quality to support project 

decisions

Project goals, data required 

to meet those goals, 

statement of what criteria 

the data must meet to be 

acceptable for project use

7-step DQOs, MQOs for all 

DQIs, statement of tolerable 

error ranges for project 

decisions

3.2.7 Special 

Training 

Requirements 

and 

Certification

To be sure that the project 

has qualified personnel to 

perform all necessary 

functions

List of specialized training 

required in the project

Matrix of special training 

needs, personnel who are 

qualified, and timeframe of 

qualifications

3.2.8 Documentation 

and Records 

Requirements

To document project 

process, outcomes, and 

supporting quality 

information, to meet clause 

7.10 of the Standards

Statement declaring where 

project documentation will 

be stored, including, but not 

limited to: QAPP, data

assessment records, interim 

and final reports

Detailed documentation and 

storage requirements per 

regulation or contractual 

obligation. These may be 

indicated in the project 

schedule as well.
QAPP Handbook A-7 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

3.3.1 Proposed Data 

Source 

Originator and 

Publication 

Information

To provide documentation 

and defensibility of data 

considered for inclusion in 

the project

Include name and date of 

data originator/origination

Publication source and peer 

review history

3.3.2 Data Format 

and 

Accessibility

To determine if the data are 

accessible for project use

Describe data format, data 

fields, and accessibility 

issues, if any

Provide SOPs for accessing 

data

3.3.3 Establishment 

of Acceptance 

Criteria

To set specific quality 

requirements for data 

brought in for project use

Establish the MQOs 

(qualitative or quantitative, 

when possible) required to 

meet project objectives 

Discuss contingencies and 

possible trade-offs if MQOs 

are not directly achievable 

with existing data

3.3.4 Sample Data 

Collection 

Methodology

To be sure data are 

appropriate for project 

purposes

State whether data were 

collected using a 

probability-based or 

judgmental sampling design

Explain the sample design 

and collection procedures 

and their implications for 

project use, such as whether 

it is reasonable to infer 

wider meaning beyond the 

original intent of the data 

collection

3.3.5 Quality 

Program and 

Quality 

Assurance 

Procedures 

Used by Data 

Originator

To document the quality of 

the existing data and to 

determine if the quality 

program is sufficient to 

support the use of these data 

for project purposes

Reference the quality 

program under which data 

were collected; if unknown, 

state the limitations on using 

data of unknown quality 

Describe results of QC 

samples included with 

existing data, data originator 

notes or reports on data 

collection, analyses, and 

limitations on interpretation 

3.3.6 Documentation 

of Sample 

Quality 

Assurance 

Procedures

To facilitate the conversion 

of existing data to a usable 

format without error.

A simple statement or 

description of how this was 

achieved

Documentation of the 

methods used and 

procedures taken to ensure 

accuracy in conversion to 

suitable format

3.3.7 Data 

Management 

Requirements

To ensure data integrity Describe data handling from 

generation, to use, to final 

storage; include copies of 

data entry forms, reports and 

description of databases; 

specify any special 

requirements for data such 

as CBI or hardware/software 

Include SOPs for data 

management, specify project 

personnel’s roles and 

responsibilities regarding 

data management, describe 

how metadata will be 

gathered and stored with 

project data
QAPP Handbook A-8 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

3.4.1 Qualitative 

Comparisons to 

Acceptance 

Criteria

To be sure existing data 

meet the qualitative 

acceptance criteria for 

project use

Describe how the project 

will assess how well data 

represent the underlying 

population of interest, and 

how comparability among 

data used in the project will 

be assessed

Include a table with a 

discussion of all qualitative 

aspects of data quality with 

MQOs for 

representativeness, 

completeness, and 

comparability

3.4.2 Quantitative 

Comparisons to 

Acceptance 

Criteria

To be sure the existing data 

meet the quantitative 

acceptance criteria for 

project use

Document the precision, 

bias, and sensitivity required 

to meet project objectives

Include a table with specific 

MQOs for the quantitative 

DQIs (precision, bias, 

sensitivity, and perhaps 

completeness) with 

discussion of contingencies 

and possible trade-offs 

between MQOs if the data 

do not fully meet the 

requirements

3.4.3 Assessments to 

Other Criteria

To be sure there are not 

other issues that will inhibit 

the likelihood of a 

successful project

State any constraints under 

which the project must 

function to be successful 

(e.g., timeframe, cost, 

attainment of specific goals)

Describe the mechanisms in 

place to ensure that these 

constraints are met

3.4.4 Interim 

Assessments of 

Data Quality

To be sure data collection is 

proceeding according to 

plan

Include statement 

encouraging project team 

members to alert 

management if they sense 

anything isn’t going quite 

right

Indicate points during the 

project timeline at which 

interim assessments of data 

quality will be conducted; 

state who will be 

responsible for their conduct 

and how they will be 

documented

3.4.5 Evaluation of 

Unconventional 

Measurements

To be sure that any unusual 

laboratory methods are fully 

documented and understood

If no unconventional 

measurement methods will 

be/were used, just state that 

here

Describe the reason for 

using unconventional 

measurement methods, 

provide SOPs for their 

implementation, and 

describe how their efficacy 

will be/was assessed

3.4.6 Evaluation of 

Unconventional 

Monitoring 

Projects

To be sure that any unusual 

sample collection methods 

are fully documented and 

understood

If no unconventional sample 

collection methods will 

be/were used, just state that 

here

Describe the reason for 

using unconventional 

sample collection methods, 

provide SOPs for their 

implementation, and 

describe how their efficacy 

will be/was assessed
QAPP Handbook A-9 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

3.5.1 Data 

Verification and 

Validation 

Targets and 

Methods

To determine if data have 

met project quality 

objectives

Provide a standard data 

verification and validation 

method or procedure that 

has been reviewed to ensure 

it meets project needs

Develop project-specific 

verification and validation 

schema that incorporate QC 

data and metadata

3.5.2 Quantitative 

and Qualitative 

Evaluations of 

Usability

To ensure that project 

decisions are supported by 

data of sufficient quality for 

project needs

State which members of the 

project team will be part of 

the evaluation of data 

usability, how it will be 

conducted and documented

Define contingencies for 

any issues that may be 

identified during the 

evaluation of data usability

3.5.3 Potential 

Limitations on 

Data 

Interpretation

To ensure that the data are 

not stretched beyond their 

appropriate use

Describe what actions will 

be taken if project data are 

deemed unusable for their 

intended project purpose

Describe how any 

limitations will be 

documented and stored in 

the metadata surrounding 

the project data

3.5.4 Reconciliation 

with Project 

Requirements

To determine if project 

requirements have been met 

Clearly state how data 

verification, validation, and 

usability results will be used 

to determine if the project 

requirements have been met; 

describe how the five steps 

of the DQA process will be 

conducted

State specific exceptions to 

statistical signficance that 

will be overturned in favor 

of “practical signficance” 

should they occur; define 

the steps to be taken for 

contingencies if the data do 

not fully support project 

requirements

3.5.5 Reports to 

Management

To document project 

outcomes

Define schedule and content 

for interim and final reports 

to management

Provide report templates

4.2.1 Title, Version, 

and Approval/ 

Sign-off

To meet the requirements of 

Clause 7.5.1 of the 

Standards for an approved 

QAPP prior to initiating 

work

Single page with title and 

required signatures

Revisions history tracking 

all changes made to the 

QAPP over time

4.2.2 Document 

Format and 

Table of 

Contents

To make it easy for the 

project team and reviewers 

to find the information they 

need

Table of contents and page 

numbers

Document control format

4.2.3 Distribution 

List

To be sure that everyone 

who needs it has access to, 

and awareness of, the latest 

version of the QAPP

Included on the title page or 

in the project organization 

section

A separate section with 

contact information and 

indication of under what 

conditions each individual 

needs to be made aware of 

revisions to the QAPP
QAPP Handbook A-10 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

4.2.4 Project 

Organization 

and Schedule

To ensure all key project 

personnel are aware of their 

responsibilities and the 

timeframe for completion

Names of key project 

personnel and period of 

performance for project

Organizational chart of all 

key project roles with names 

and Gantt or similar tracking 

chart of schedule

4.2.5 Project 

Background, 

Overview, and 

Intended Use of 

the Model

To provide sufficient 

information as a foundation 

for project goals and to 

clarify the expected uses of 

the data

General overview and 

statement of how data and 

models will be used in this 

project

Detailed project history, 

references to other 

background documentation, 

regulatory basis for data use

4.2.6 Data/Project 

Quality 

Objectives and 

Acceptance 

Criteria

To ensure that data for 

models collected, or used in 

the project, are of sufficient 

quality to support project 

decisions

Project goals, data required 

to meet those goals, 

statement of what criteria 

the data must meet to be 

acceptable for project use

7-step DQOs, MQOs for all 

DQIs, statement of tolerable 

error ranges for project 

decisions

4.2.7 Special 

Training 

Requirements 

and 

Certification

To be sure that the project 

has qualified personnel to 

perform all necessary 

functions

List of specialized training 

required in the project

Matrix of special training 

needs, personnel who are 

qualified, and timeframe of 

qualifications

4.2.8 Documentation 

and Records 

Requirements

To document project 

process, outcomes, and 

supporting quality 

information, to meet clause 

7.10 of the Standards

Statement declaring where 

project documentation will 

be stored, including, but not 

limited to: QAPP, data 

packages, assessment 

records, interim and final 

reports

Detailed documentation and 

storage requirements per 

regulation or contractual 

obligation. These may be 

indicated in the project 

schedule as well.

4.3.1 Data 

Requirements 

for Model Input 

To ensure that appropriate 

data are available for use in 

the model

Describe the type, quantity, 

and quality of data required 

for use in the model, and 

how it will be identified or 

collected

Specify data collection or 

production and analysis 

requirements for the model 

input data (per Chapters 2 or 

3 of this QAPP Guidance)

4.3.2 Data 

Management 

Tasks

To ensure the integrity of 

data used in the project

Discuss how data will be 

collected, produced, 

managed, and stored

Include SOPs for data 

management

4.4.1 Model 

Evaluation 

Methods and 

Activities 

To know whether or not the 

model and its results are of 

sufficient quality to meet 

project objectives

Describe how the project 

will evaluate that the model 

is based on documented

science, appropriate data, 

performs adequately, and 

meets project objectives

Include specific model 

evaluation plans throughout 

the project and model’s 

lifecycle, including factors 

that would trigger model 

modification
QAPP Handbook A-11 Agency Review Draft 

Section 

Number

Title Why Important Simple Case Complex Case

4.4.2 Evaluation of 

the Model

To be sure the model 

functions as needed for 

project purposes

Documentation of how the 

model has been and will 

continue to be assessed for 

its ability to provide output 

that meets project objectives

Calibration and internal 

quality check logs, model 

change/development logs, 

evaluation criteria and 

identification of responsible 

staff

4.4.3 Model Output 

Sensitivity and 

Uncertainty 

Analysis

To determine if the model is 

providing sufficient 

certainty to reach a decision 

with the desired level of 

quality

List anticipated outputs of 

sensitivity and uncertainty 

analyses, how they will be 

documented, and any 

criteria for their 

acceptability

Computer code for the 

uncertainty and sensitivity 

analyses, documentation of 

the criteria that would lead 

to refinement of the model 

via the path identified in the 

sensitivity analysis

4.5.1 Peer Review To ensure the validity of the 

science behind the 

development of the model

Peer Review by colleagues 

familiar with the problem

Formal Agency Peer 

Review with documented 

charge and response

4.5.2 Description of 

Model 

Documentation

To provide defensibility and 

reproducibility

Description of how model 

selection or development, 

modifications, input data, 

results, sensitivity and 

uncertainty analyses, and 

evaluation will be 

documented

SOPs for document control 

and archiving of 

programming code, 

software, and related 

manuals

4.5.3 Specifications 

for Model 

Maintenance 

and User 

Support

To ensure the model doesn’t 

cease to function properly 

during the project lifecycle

Simple statement of the 

intent to maintain the model 

throughout the project 

lifecycle

Specific schedule for model 

maintenance and SOP or 

description of user support 

to be provided throughout 

the project lifecycle

4.5.4 Reports to 

Management

To document project 

outcomes

Define schedule and content 

for interim and final reports 

to management

Provide report templates
QAPP Handbook B-1 December 2012 Agency Review Draft

APPENDIX B

DATA QUALITY INDICATORS

The choice of MQOs for selection of Data Quality Indicators (DQIs) is critical to evaluating and, 

in some cases, controlling environmental data quality. The criteria that will be used to gauge 

measurement performance should be determined and documented for each matrix, analytical 

group, concentration level, and analyte, if applicable. DQIs relate to the parameters of precision, 

accuracy/bias, representativeness, comparability, sensitivity, and completeness. It is extremely 

important to recognize that data quality is a function of both the sampling and measurement 

processes; it is not solely equated to the quality of individual analytical results. To simplify the 

way data quality is examined, and to facilitate communication about data-quality attributes, the 

following DQIs important to environmental studies will be defined: precision, bias, 

representativeness, comparability, completeness, and sensitivity (PBRCCS). These six DQIs 

used to be referred to by the acronym PARCCS, with the "A" in PARCCS referring to accuracy

instead of bias. Accuracy is actually comprised of random error (precision), and systematic error 

(bias), and these indicators are discussed separately and hence the “A” has been replaced by “B”. 

The goal of this appendix is to familiarize the reader with the types of DQIs, how each can be 

used to assess data quality and usability, and how they are measured. 

DQIs for precision, bias, and sensitivity can be defined and MQOs measured in quantitative 

terms; MQOs for representativeness, comparability, and completeness can be defined and 

measured in both qualitative and quantitative terms. Establishing performance criteria for 

representativeness is a principal element in setting goals for the sampling design. Establishing 

performance criteria for sensitivity sets quantitative goals for the quality of data generated in the 

analytical measurement process. Establishing performance criteria for precision and bias 

potentially sets quantitative goals for the quality of data generated in both the sampling and 

analytical measurement process.

Table B-1 Relationship Between Quality Terms

DQOs or PQOs Qualitative and quantitative quality objectives for project conclusions or decisions. The 

decision to call these Data Quality Objectives (DQOs) or Project Quality Objectives (PQOs)

depends on the organization’s preferences.

DQIs These are the indicators of data quality attributes.

MQOs or MPCs Acceptance thresholds or goals for the data, usually based on individual DQIs. The decision 

to call these Measurement Quality Objectives (MQOs) or Measurement Project Criteria 

(MPCs) depends on the organization’s preferences.

Measurement Quality Objectives (MQOs) should be developed as an integral part of the 

sampling and analysis design generated during systematic planning (e.g., in Step 7 of the DQO 

process) and see also Annex C of the two Standards. MQOs are not synonymous with project 

DQOs. DQOs establish the full set of specifications for the design of the data collection effort. 
QAPP Handbook B-2 December 2012 Agency Review Draft

The design typically incorporates and specifies requirements for total variability. These 

requirements are used to establish performance criteria, stated as MQOs (MPCs), for significant 

components of the total variability. It is emphasized that sampling and analytical procedures 

should be selected after the performance criteria have been established. The relationship among 

these quality terms (DQOs/PQOs, DQIs, and MQOs/MPCs) is presented in Table B-1. The 

historical focus of DQIs has been on the laboratory measurement processes, but, as discussed 

later, DQIs can be chosen to capture the effects of the full measurement system, sampling design, 

and sample collection processes, all of which affect the overall quality of study data. Sections

B.3 through B.9 detail the DQIs and specific issues relating to each. Table B-2 offers a summary

of the definitions for data quality attributes commonly monitored with DQIs along with brief 

descriptions of methods for determination of those attributes.

Table B-2 Data Quality Indicators

DQI Definition Examples of Determination

Precision An evaluation of agreement among 

replicate measurements of the same 

property under similar conditions; also 

referred to as random error or 

measurement variability and usually 

expressed as standard deviation, 

variance, percent difference, or range, in 

either absolute or relative terms

Overall project precision is measured by 

collecting data from collocated field duplicate (or 

replicate) samples. Precision specific to the 

laboratory is measured by analyzing laboratory 

duplicate (or replicate) samples

Bias The systematic or persistent distortion of 

a measurement process resulting in error 

in one direction

Measurement of materials with a known 

concentration (e.g. performance evaluation or 

reference materials), analysis of matrix spikes, or 

the use of laboratory control samples

Accuracy A measure of the closeness of an 

individual measurement to a known or 

reference value; includes a combination 

of random error (precision) and 

systematic error (bias) components of 

both sampling and analytical operations

Replicate analysis of a reference material or reanalyze a sample to which a material of known 

concentration or amount of pollutant has been 

added; usually expressed either as percent 

recovery or as a percent bias

Representativeness A qualitative measure of the degree to 

which data accurately and precisely 

represent a characteristic of a population 

parameter 

Evaluation of whether a sample that is collected

and then processed and sub-sampled by the 

laboratory is proportionately representative of 

some predefined population characteristic or 

property. As such, representativeness is an 

“objective-defined” parameter (e.g. total 

concentration versus dissolved concentration 

versus bioavailable concentration)

Comparability A qualitative term describing the degree 

to which different processes, methods, or 

data agree or can be represented as

similar. It describes the confidence that 

two datasets can contribute to a common 

analysis and interpolation. Comparability 

criteria must be determined for each 

matrix, analytical group, concentration 

level, and analyte (if possible).

A comparison of the output of two sediment 

transport models via sensitivity analysis. Or 

comparison of the sample collection methods, 

analytical procedures, holding times, stability 

issues and QA protocols. One study with results 

in ug/L is not necessarily comparable to another 

with results in ppb. A similar argument exists 

between wet and dry weight comparisions
QAPP Handbook B-3 December 2012 Agency Review Draft

DQI Definition Examples of Determination

Completeness An evaluation of the amount of data 

needed to be obtained from a 

measurement system; expressed as a 

percentage of the number of 

measurements that should have been 

collected or were planned to be collected

Evaluation of the number of measurements 

needed to make a determination of the project 

results and comparison of this to the number of 

samples planned to be collected

Sensitivity The capability of a method or instrument 

to discriminate the parameter of interest

at the level of interest. Terms sometimes 

used to describe sensitivity include 

Method Detection limit (MDL), Limit of 

Detection (LOD), and Limit of 

Quantitation (LOQ)

The measurement responses representing 

different levels or amounts of the variable of 

interest, MDL study, and verification of LOD 

B.1 SOURCES OF MEASUREMENT UNCERTAINTY

Before discussing details of DQI calculations and uses, a presentation of the framework used in 

this document for examining error sources is necessary. Consideration of all the sources of error 

is critical as specific DQIs can be established to address specific sources of error. Total study

error is a measure of the uncertainty in a metric such as a site mean concentration caused by the 

combination of all error sources in the study design. The term “total” is relative to the overall 

boundary conditions (spatial or temporal boundaries) for which a decision will be made 

(sometimes referred to as the decision unit) or study conclusions drawn, based on the relevant 

metric. These boundaries are defined in the DQO process and may include the overall population 

of interest, as well as specific subpopulations for which a decision will be made. A sampling unit 

is the portion of the physical environment from which one or more samples may be taken, 

resulting in measurements appropriate for an intended use. The term "error sources" refers to any

factors that increase uncertainty in a measured value. Generally speaking, these error sources are 

the result of natural variability in the sampled media and uncertainty associated with the 

sampling and measurement process. The framework for examining study errors will also serve as 

a context for clarifying the relationship among the total study error, DQOs, and MQOs.

The contribution to total error from within-unit sources is largely a function of the boundary

conditions for the study at hand, the inherent variability of the characteristic of interest within the 

unit, the difficulties associated with obtaining a specimen from the unit, and the error associated 

with the measurement process itself. 

B.2 ESTABLISHING MQOs IN THE CONTEXT OF DQOs

One way to employ DQIs is as a means of specifying MQOs which, if achieved, will provide an 

indication that the resulting data are expected to meet the DQOs. This is not to say that simply

achieving MQOs for the measurements ensures that the DQOs are met. Typically it is more 

important to ensure that an adequate number of samples, with appropriate sample support, are 

collected to represent the population of interest. If this is the case, then ensuring that MQOs were 

achieved provides an indication that the measurement quality will be adequate for the intended 

use. Used in this way, DQIs provide a metric against which the performance of a program can be 

measured during the implementation and/or assessment time frames. 
QAPP Handbook B-4 December 2012 Agency Review Draft

During the design phase, the type and number of samples required to achieve DQOs and the way

in which these samples should be optimally allocated across space and time are developed. 

Guidance on Choosing a Sampling Design for Environmental Data Collection, EPA QA/G-5S, 

(EPA 2002a) provides guidance on alternative design approaches. For statistically-based 

sampling designs, an estimate of total study error is required to determine the appropriate 

number of samples required to achieve a certain performance goal, In some cases, historical data 

and information are available, but were collected with older, or different, sampling or analytical 

methods than are currently considered (comparability). In other cases it may be advisable to 

derive an estimate of variability using other information on the background of the measured 

variable. Usually this involves the use of statistical theory combined with practical knowledge.

To determine the potential impact of using newer methodologies on total study accuracy

(precision and bias), it is useful to see how much the error sources of variability contribute to the 

total study variability. If the newer methods are more accurate, specific MQOs can be established 

based on the expected improvements in laboratory performance, and these improvements can be 

factored into the design equation. It is emphasized that the performance goals of precision (and 

the rest of the MQOs) be considered relative to action levels for each analyte. For example, if the 

newer methods are less accurate, but the concentrations are well above (or below) their action 

levels such that a decrease in accuracy or loss of precision will not affect the ability to make an 

appropriate decision, then it may still be optimal to use them. If heterogeneity is problematic,

then obtaining a larger number of samples would increase coverage (representativeness) to better 

represent the population of interest and account for between-unit variability. 

A practical approach to accomplish this is to first determine the performance required for the 

DQI, based on an analysis of the relative impact of a specific error source or quality attribute on 

total study design. This will be based on an understanding of what method or procedure precision 

levels are tolerable, given a particular design (type, number, and allocation of samples) under 

consideration. The design process involves evaluating tradeoffs between the sampling aspects 

and measurement aspects and should be aimed at achieving an optimal combination of the two

relative to the DQOs or project objectives. 

If laboratory or method defaults are not adequate or if alternatives are much less expensive, then 

MQOs should be specified and a new agreement reached with the laboratory performing the 

analyses (or a new instrument or method selected) to meet or exceed these more stringent 

requirements. If aimed at the analytical laboratory, this may take the form of a special analytical 

request or a performance-based method contract. If aimed at the use of an instrument or method, 

it may take the form of a new standard operating procedure designed to ensure that the MQO is 

achieved such as SOPs for replicating X-ray fluorescence (XRF) measurements to achieve a 

precision requirement. MQOs should be driven by the DQO process and should always be 

defined before listing laboratory or method "default" limits in a QAPP. This is especially 

important when the project does not require levels as stringent as the laboratory’s default limits 

to be adequate for the intended use. For example, if default requirements for sensitivity are far 

better than required to support a particular decision, project-specific, less restrictive requirements 

can be stated in the QAPP. 
QAPP Handbook B-5 December 2012 Agency Review Draft

B.3 PRECISION

Precision is a DQI upon which statistically designed sampling is based. For this reason, precision 

is among the most important DQIs in environmental projects. Precision is the measure of 

agreement among repeated measurements of the same property under identical or substantially

similar conditions. High precision refers to close agreement among repeated measures, whereas 

low precision refers to poor agreement among repeated measures. For environmental studies, the 

property measured is typically a concentration of a contaminant, but any physical measurement 

has error; therefore, the precision of that measurement can be examined. Random errors or 

fluctuations in the measurement process always result in some range of values of these repeated 

measurements. Precision is a quantitative indicator of the dispersion generated from these 

random errors.

Precision is estimated by using some form of replication (e.g., collocated field duplicates, field 

splits, laboratory splits, matrix spike duplicates, instrument replicates) followed by calculation of 

a DQI based on the replicate measurements. Estimates of precision provide a measure of 

agreement among replicate analyses; MQOs for precision set tolerable limits of imprecision for a 

study. 

One of the most common precision DQIs are the summary statistics range for duplicate 

measurements and standard deviation for multiple measurements. One nice feature of these 

statistics is that their units are the same as the individual measurements, and they are relatively 

easy to calculate. Another widely used measure is relative percent difference, usually used when 

only two samples are taken.

B.4 BIAS

Bias is systematic or persistent distortion of a measurement process that causes errors in one 

direction. Bias may originate from sources such as calibration errors, response factor shifts, 

unaccounted-for interferences, or sample contamination. The sample itself may generate real or 

apparent bias caused by a matrix effect or variation in physical properties such as particle size. 

DQIs for precision are quantitative indicators, which reflect the magnitude of systematic error 

resulting from these effects. Bias can be in the positive (a false positive or elevated 

signal/concentration) or negative (a false negative or suppressed signal/concentration) direction 

from the true value. 

The difference between the measured and known or expected result is a DQI for bias. For spikes 

and reference materials, bias is most conveniently expressed as a fractional or percent 

comparison of the measured result to the expected result. Relative bias is also a common 

indicator, which indicates both the magnitude and direction (positive or negative) of the bias. 

Expected results are based on the known properties of the QC sample. 

The expected result is usually the true value derived from a standard or a theoretical value and

documented in the project’s QAPP. 

measured result - expected result

relative bias = expected result 
QAPP Handbook B-6 December 2012 Agency Review Draft

Relative bias is the ratio of the difference between the measured and expected results to the 

expected result. Relative bias is sometimes used to describe bias in terms analogous to relative 

precision. For applications of bias with a known contaminant, a commonly used measure of bias 

is percent recovery, which is simply the percent of the expected result that is measured. Percent 

recovery calculated from matrix spikes is the difference between the spiked and unspiked sample 

results divided by the spiking amount. A completely unbiased result thus has a recovery of 1 (or 

100%).

Minor short-term problems that do not create a systematic bias are best reflected in precision 

DQIs (random errors). For example, if an analyst makes an error on a dilution as part of the 

sample preparation step, that mistake will bias the individual result. However, this sort of error is 

part of the overall precision in the measurement process. Because the analysis of QC samples 

(see Appendix C) includes the entire measurement step, the precision of the measurement 

process is also imposed on results intended for estimating bias. For this reason, estimates of bias 

primarily serve as a quality control function during implementation of a project plan. During the 

planning phase, procedures put in place to minimize the potential for bias and the use of 

estimates of bias to verify the procedures should have the desired effect of keeping bias to a 

minimum and assessing the potential effects on project objectives. 

Procedures for estimating bias are also valuable tools for ensuring comparability of data. 

Completeness of a data set also has the potential to impact bias. If a data set is incomplete, any 

systematic trend to the missing data may cause bias in estimates of population parameters based 

on this faulty data set. 

B.5 ACCURACY

Accuracy is a measure of the overall agreement of a measurement to a known value. In a limiting 

case where random errors are very tightly controlled, bias dominates the overall accuracy. In 

general, however, both precision and bias contribute to accuracy. A measurement result with 

zero bias may not be accurate if the measurement process is not precise. Figure B-1 demonstrates 

how different combinations of precision and bias can contribute to accuracy. The true mean 

value, indicated by the dashed line, is 10.0 and the red line represents a hypothetical action level 

of 12.0 that might be used as a “bright line” or limit for an action. Diamond symbols indicate the 

observed measurements. Part A of Figure B-1 shows the least desirable situation of positive bias 

(a large shift in the positive direction) and low precision (a large spread in the results). The 

accuracy and precision of these data are poor. Part B of Figure B-1 shows the case of low bias 

(no apparent shift in a positive or negative direction) and low precision. Any one of the 

measurements is not accurate, but an average of all the measurements would be accurate relative 

to the true mean. Part C of Figure B-1 shows data with positive bias and high precision. The 

accuracy of these data is poor. In this case, random errors are well controlled, but a systematic 

error limits the accuracy of each individual measurement. Part D of Figure B-1 shows the case of 

low bias and high precision. Each individual measurement is accurate. Notice that in C and D the 

DQI of precision is critical for making a correct decision relative to the action level. It is obvious 

from this visual example that both precision and bias be considered relative to the action level in 

order to assess whether the DQI has a significant impact on the ability to make a correct 

decision. Whenever an action level is “close” to the measured data from the site (relative to the 

dispersion of the data), the more critical the DQIs are. 
QAPP Handbook B-7 December 2012 Agency Review Draft

Figure B-1 Influence of Bias and Precision on Accuracy

B.6 REPRESENTATIVENESS

Representativeness, as it was defined by ANSI/ASQ E4-2004 (ANSI 2004), is "The measure of 

the degree to which data accurately and precisely represent a characteristic of a population, 

parameter variations at a sampling point, a process condition, or an environmental condition." 

This definition of representativeness encompasses issues at both the micro- and macro-scale by 

addressing both how well measurements taken within a sampling unit reflect that unit (parameter 

variations at a sampling point) and the degree to which measurements from a set of sampling 

units represent (allow you to make inferences about) the population of interest (accurately and 

precisely represent a characteristic of a population). Central to representativeness is assurance 

that both the sampling and measurement processes are free from known biases. Developing a 

clear understanding of the "population" that is the subject of an experiment or investigation is the 

key to assessing representativeness. The characteristics of the population include the subject's 

identity or class (e.g., the particular property that is to be measured), the spatial distribution of 

the property, and in some cases, the temporal characteristics of the property. Step 4 of the DQO 

process (EPA 2006a) explicitly focuses on establishing a clear definition of the population of

interest and any subpopulations that data should adequately represent to support decisions at the 

desired scale. Representativeness is usually considered a qualitative term that does not lend itself

to measurement by an MQO. 
QAPP Handbook B-8 December 2012 Agency Review Draft

Instead, sampling precision and bias indicators related to the desire to represent environmental 

phenomenon at specified spatial and temporal scales are used to control the quantitative aspects 

of representativeness. 

Representativeness was established as a DQI as a result of the recognition that characteristics of 

interest to environmental problems are heterogeneously distributed in space and time within the 

environment, and that careful attention should be paid during planning and implementation of a 

study to ensure that a set of samples adequately mirrors or reflects the characteristics of interest. 

Project managers and decision makers want assurance that the results of a particular data 

collection effort are not biased in any known way by the sampling or analysis design. Such a 

bias, combined with the inherent heterogeneity in the environment and uncertainty in the 

measurement process, could result in an incorrect conclusion. Lack of representativeness can 

have a direct impact on the ability to make the correct decision when relying on data. To ensure 

representativeness, careful attention during the entire life cycle of a project is required. 

The sections on precision and bias covers in detail some of the more quantitative elements of 

representativeness. Central to representativeness is assurance that both the sampling and 

measurement processes are free from known biases. Representativeness is widely used in a less 

precise manner in the environmental community. For example, representativeness is a word 

commonly used to mean:

 there is an absence of biasing forces;

 it is a miniature or replica of the population;

 it is a typical or ideal case;

 there is a wide coverage of a population; 

 it permits good estimation;

 it is good enough for the purposes of the study; or 

 a statistically-based sampling method was used. 

While a number of these are indeed characteristics of a representative study, a more careful 

definition will better delineate suitable indicators of representativeness. 

The concept of representativeness is called out in a number of EPA regulatory programs. For 

example, the Resource Conservation and Recovery Act (RCRA) talks about “...expected to 

exhibit the average properties of the universe or whole” (RCRA CFR 260.10). The Toxic 

Substances Control Act (TSCA) refers to “...at locations representative of the air entering the 

abatement site” (TSCA 40 CFR 763). In the air programs we see a discussion of 

representativeness: “...should be selected on the basis of spatial and temporal representativeness” 

(40 CFR 51 Appendix W). The wastewater program simply states, “...samples should be 

representative of daily operations” (40 CFR 403.12(b)). While these statements are lacking in a 

rigorous definition, they provide some understanding of the importance of this indicator to EPA 

programs.
QAPP Handbook B-9 December 2012 Agency Review Draft

The process involved in obtaining representative samples includes planning, implementation, and 

assessment. In addition to sample design, careful attention should be paid to the sample 

collection, measurement and analysis processes. 

Representativeness also has relevance in the world of laboratory research studies and 

experimental design. For a discussion of these issues, refer to texts on experimental design, 

including Box, Hunter, and Hunter (1978) and Cochran and Cox (1957).

To understand the importance and scope of this DQI, imagine just these few examples where 

representativeness is not achieved and consider the potential consequences:

 surface water samples are taken from the stagnant water nearest the shore in a calm pool, 

and are then meant to represent the characteristics of the fast-flowing river;

 soil samples are collected using a technique that preferentially selects the smallest 

particles and omits the larger particles, thus under-representing those with lower surface 

area proportionately, and therefore lower levels of externally adhering contaminants; or

 a survey of public perception to off-shore oil drilling conducted entirely in Ohio, Kansas, 

and Nevada, and presented as national opinion.

Representativeness, as a DQI, is most relevant when viewed in the context of the data’s intended 

use. Representativeness between sampling units and within sampling units should be considered, 

and qualitative MQOs for both should be established and documented in the QAPP.

B.7 COMPARABILITY

Comparability is the qualitative term that expresses the confidence that two datasets can 

contribute to common interpretation and analysis of the parameter or matrix of interest. 

Quantitative measures of comparability are also available involving statistical tests that measure 

the similarity or difference between two or more datasets. Comparability should be carefully 

evaluated in order to establish whether two datasets can be considered equivalent in regard to the 

measurement of a specific variable or groups of variables (EPA 1997, EPA 2010a).

Comparability is a very important qualitative DQI for analytical assessment and is critical when 

considering the combination of datasets with the same analytes. The assessment of this DQI

determines if analytical results reported are equivalent to data obtained from similar analyses. 

Only comparable datasets can be readily combined and used for subsequent statistical 

evaluations. For example, if two separate investigations of the same site utilized different 

analytical methods with different sensitivities, combining the data may violate the underlying 

assumptions of which the statistical tests depend. The potential differences in precision and bias 

could be misinterpreted as within-unit sampling error.

As with any decision about the usability of data, it is important to consider the decision that the 

data are meant to support. Separate determinations of the comparability of datasets may be 

necessary for each decision the data are used to support. Examples in the uses of comparability

are found in Chapter 3, Section 3.4.1.
QAPP Handbook B-10 December 2012 Agency Review Draft

B.8 COMPLETENESS

Completeness is a measure of the amount of usable data obtained from a measurement system, 

expressed as a percentage of the number of measurements that should have been collected 

according to the study design (i.e., measurements that were planned to be collected). Percent 

completeness is calculated using the formula: 

Percent Completeness = (number of usable measurements) x 100 % 

(total number of measurements planned)

Completeness is one measure of how well a sampling and analysis design was implemented. A 

data set that is 100% complete is the result of careful planning and precise implementation of the 

data collection plan. Completeness is not intended to be a measure of representativeness; for 

example, it does not describe how closely the measured results reflect the actual concentration or 

distribution of the pollutant in the media sampled, but it may be a contributing factor. A 

complete data set may or may not achieve the project objective depending on how well the 

sampling plan reflects the conceptual model, how accurate the conceptual model was prior to 

sampling, and whether the distribution of the reported data are similar to those anticipated. 

When considering completeness, attention should be given not only to simply complete samples 

but also to the measurements on analytes within a sample. If, for example, measurements on 6 

analytes were requested, but only 4 actually made, this sample would be classified as 

“incomplete” if the missing analyte measurements were part of the MQOs or DQOs. It follows 

that for some instances, completeness should be defined in terms of analyte/sample combinations

rather than samples. For example, suppose measurements of six analytes were requested in 20 

samples, but in one sample only four were actually performed or found acceptable readings; the 

entire sample would be found “incomplete” with a completeness of 19/20 = 95%. If 

completeness is defined in terms of analyst/sample combinations, then the completeness would 

be 118/120 = 98.33%.

The important question for decision makers is whether the number of measurements is sufficient 

to support the decision to be made. For example, there could be only 70% data completeness 

(30% lost or found invalid), but, because of the nature of the study design, the results could still 

be representative of the target population and yield valid estimates. Conversely, a data set with 

much higher completeness but systematically omitted or rejected data may be insufficient to 

yield valid estimates of the parameters of interest. Key questions for the DQI of completeness 

would include: 

 Was the number of field samples taken for each matrix and analytical group the same as 

that documented in the sampling plan?

 Were all the associated quality control samples taken?

 Were all the samples delivered to the lab analyzed?

 Were the samples analyzed with the appropriate method as documented in the QAPP?

and

 Were any of the sample results rejected due to quality concerns?
QAPP Handbook B-11 December 2012 Agency Review Draft

It is suggested that the QAPP set a quantitative goal for completeness. If that goal is achieved, 

then no further consideration of this DQI is necessary. If the completeness goal is not achieved, 

the QAPP should make clear how the project team will proceed and what options might exist. It

is important that the completeness goal is potentially attainable (95% when using only 10 

samples, for example) and that it is linked to the project’s objectives. It is important to 

distinguish between incompleteness of investigative samples and QC samples, as the loss of the 

latter can be of less importance than the former.

B.9 SENSITIVITY

Sensitivity generally refers to the capability of a method or instrument to discriminate between 

small differences in analyte concentration. Both the precision of the instrument and the slope of 

the calibration curve limit sensitivity. Chemists typically define sensitivity as the slope of the 

calibration curve at the concentration of interest (Skoog 1985, Currie 1995). If two methods have 

equal precision, the one having a steeper calibration curve will be the more sensitive. Sensitivity

can also be evaluated from the standard deviation of replicate analyses at any concentration level 

or can be evaluated from the confidence bound on a calibration curve. It represents the minimum 

difference in two samples that can be distinguished with a defined confidence (Taylor, 1987).

The sensitivity indicators of primary interest relate to limits of detection. The detection limit

(DL) is generally considered to be the minimum true concentration of an analyte producing a 

non-zero signal that can be distinguished from the signals generated when no concentration of 

the analyte is present, with an adequate degree of certainty. There are a plethora of different 

approaches to the determination of detection limits that can be broadly categorized into two 

groups: those that evaluate the statistical variability of instrument responses with and without the 

analyte present. There are several critical aspects of determining detection limits that have 

historically been overlooked:

 the variability of instrument responses are commonly heterogeneous or heteroscedastic 

(i.e. standard deviation is not linear over the concentration range);

 most common methods only consider false positives (type I or α error) at the Detection 

Limit (DL), ignoring the importance of false negatives (type II or β error) (The false 

negative rate at the DL is 50%. Therefore, there is a 50% chance of error when reporting 

a sample result as <DL (or <MDL) when nothing is detected);

 the distribution of measured values around the true detection limit is assumed to be 

normally distributed. (In reality that assumption at the true DL is likely false. 

Unfortunately we cannot verify that assumption because we do not have statistical 

confidence at the DL);

 instrument responses, variability, and subsequent detection limits are greatly affected by 

the sample matrix, instrument conditions, and even the analyst; and

 many analytical methods produce nonzero signals even when a target analyte is not 

present. 

The Limit of Detection (LOD) is the smallest concentration of analyte present in a sample in 

order to be detected at the 99% confidence level. The LOD differs from the DL in that the 

probability of a false negative is 1%. The LOD is usually >1 to 2 times the DL. 
QAPP Handbook B-12 December 2012 Agency Review Draft

The quantitation limit (QL) is another concept that is very important for environmental decision

making. The QL relates to the lowest concentration at which the method is expected to be able to 

quantify the amount of analyte present in the sample. The QL is usually 5 to 10 times the DL. 

The Limit of Quantitation (LOQ) is the smallest concentration with a specified level of precision 

and bias. That is, samples with concentrations at or above the LOQ can be quantified with a 

known level of accuracy. Detection and quantitation levels (DLs, QLs, LODs, and LOQs) vary

by analyte and by matrix and often vary among laboratories. It is critical that when referring to 

these levels the specifications are made very clear.

Analytical capabilities are constantly improving, resulting in greater sensitivity and lower 

detection limits. This improvement in analytical capabilities is frequently the vehicle that drives 

regulatory and, hence, project requirements. Investigators often base their sensitivity

requirements upon analytical method capabilities rather than upon project-specific objectives. 

The problem for a project team is to determine the levels of sensitivity that will generate data 

adequate for decision making, to establish MQOs based on this evaluation, and to be sure that the 

indicator of sensitivity used to evaluate a particular method appropriately reflects the 

performance of the method in the particular matrix of interest. MQOs tie the required 

measurement quality to the project DQOs. 

The project team should always consider the needed sensitivity of a measurement prior to 

requesting laboratory analyses. Once the needed sensitivity is determined, the project team can 

work with laboratory personnel to choose the appropriate analytical method. Sensitivity is 

dependent on the sample matrix, the analytical method, the instrument used, and the operator 

conducting the analysis. The sensitivity of the analyses potentially impacts data usability for a 

given decision. A couple of issues that are frequently encountered in environmental projects 

relating to sensitivity are described below, along with suggestions for handling them. 

Project action levels are numerical values used by the decision-makers as a bright-line for 

selecting one of two or more alternative actions (e.g. remediate or leave in place). Action levels 

may be regulatory thresholds such as a Maximum Contaminant Level (MCL), a risk-based or 

toxicity reference benchmark, or a treatment standard. Some of these action levels may be at 

concentrations well below the DL of many analytes. However, because of the great uncertainty at 

the DL, the QL should always be less than project action levels. During project planning, project 

teams may be faced with the dilemma of targeting action limits at or below QLs. 

It is recommended that the laboratory perform project-specific confirmation of action limits in 

each matrix of concern. Some types of DLs such as MDLs are determined by the laboratory

under ideal conditions using a clean reference matrix. Analytical sensitivity is impacted by the 

presence of interferences. Therefore, the QAPP may include a requirement to perform a 

sensitivity verification study for each matrix. The laboratory can verify if the project action 

levels can be met by spiking the appropriate matrix at the action level for each analyte. The 

verification procedure for establishing the ability to meet the action levels must empirically

demonstrate quantifiable estimates of precision and bias at the action level. When precision and 

bias do not meet project requirements at project action levels, several options should be 

considered to ensure that the QLs are below the project action levels:
QAPP Handbook B-13 December 2012 Agency Review Draft

1. Modification of the proposed method. In some cases, it may be possible to modify the 

proposed method to achieve improved sensitivity. Possible modifications that could have 

a beneficial impact on sensitivity include:

 extracting more sample volume or mass, or decreasing the final extract volume

(bearing in mind that problems in homogeneity and interferences may result in an 

increase in bias); 

 including additional cleanup steps (sample preparation steps to eliminate 

interferences); or

 use a different wavelength or ion for quantitation.

Performing analyses in selected ion monitoring mode rather than full-scan mode during 

mass spectrometry analysis is an example of a modification to the method that may 

improve sensitivity.

2. Use a different method with better sensitivity. For many environmental analysis needs 

(e.g., measuring the concentration of polycyclic aromatic hydrocarbons in sediment), 

multiple analysis methods are available. The differences between methods may include 

cost, availability, and sensitivity. If the project action level is below or near the QL for a 

particular method, it may be possible to identify and implement a different method for 

analysis for which the QL is well below the project action level. For example, if running 

inorganic analyses using Inductively Coupled Plasma Optical Emission Spectroscopy 

(ICP-OES), is not achieving the desire level of sensitivity, switching to ICP-Mass 

Spectroscopy (ICP-MS) analyses might provide the additional sensitivity to meet 

objectives.

3. Consider alternative action limits when it is reasonable to question the applicability of a 

pre-set action level for a particular project. For example, taking action at the level of the 

MCL if there is no potential for the water to be studied (e.g., water in an ephemeral 

stream) to be used as a drinking water source may be overly conservative, and other, 

more project-specific action limits could be considered.

4. Report all analytic results with their associated uncertainties (precision and bias).

Although the uncertainty of results below QLs is relatively large, as long as the 

uncertainty is known a determination can be made regarding the usability of the data in 

relation to the project action level. If all results are reported with their associated 

uncertainties, the project team will be able to ascertain the level of confidence that would 

be achieved if they make a decision regarding an action limit that is below the QL. One 

way to establish the uncertainty (precision and bias) in the estimated results is using a 

technique developed by Hubaux and Vos (1970). The Hubaux-Vos method is an 

extrapolation method that utilizes the linear regression of the calibration curve. The DL is 

determined by the width of the confidence bands of the calibration process. This 

approach has the added benefit of considering both false positive and false negative 

instrument readings. If this approach is utilized, the detection limits should be reported 

along with the total number of calibration points, the number of replicates, and the 

associated error rates used to define the precision and bias bounds. 
QAPP Handbook B-14 December 2012 Agency Review Draft

Another issue relating to sensitivity is the way that very low results are reported as there is a 

great deal of uncertainty associated with very low results. The high uncertainty makes it difficult 

for chemists to be able to state what concentration of an analyte is present in the sample. One 

way that chemists handle this is by reporting that the result is less than some detectable or 

quantifiable amount. There are several different conventions for how this is done such as:

 choosing protection against false positive results (incorrectly reporting that an analyte is 

present in a sample when in reality it is not present) such as:

o results below the MDL are reported as undetectable ( < DL),

o results between the MDL and the practical quantitation limit (PQL) are reported 

as estimated), and 

o results above the PQL are reported directly.

 Choosing protection against false negative results (incorrectly reporting that an analyte is 

not present in a sample when it actually is present) such as:

o all results below the laboratory DL are reported as undetectable (as <LOD, where 

LOD is one or more times the DL),

o results between the DL and LOQ are reported as estimated, and 

o results above the LOQ are reported directly.

With multiple reporting conventions in use, many different methods for setting the DL (EPA 

2010), and many different methods for establishing the QL, it is clear that Project Managers need 

to document in the QAPP which convention they will adopt. If the standard practices of the 

laboratory are deemed acceptable for the project, it should be stated in the QAPP and the 

laboratory SOPs relating to their DLs, QLs, and reporting of results should be summarized 

and/or attached. If the DL is going to be calculated using a matrix-specific method (e.g., Hubaux 

and Vos 1970), then the methods should be well documented in the QAPP.

Finally, while setting a sensitivity level that must be reached to attain DQOs would appear to be

straightforward, it may not be. Is it the DL or the QL, or both that need to be defined in MQOs?

Is a reporting convention that protects only against false positives acceptable for the needs of the 

project? Does the project call for a reporting convention that protects against false negatives? In

the arena of DLs, QLs and reporting conventions for data at very low levels, there is not an 

overall best way to proceed as there are just a few common practices that have become relatively

standard for environmental data analyses. If DQOs indicate that these issues might be important 

(i.e., the concentration levels at which decisions must be made are very low), it is suggested that

the project team carefully consider how MQOs for sensitivity are defined, and how the 

laboratory statement of work is written to ensure that the MQOs can be achieved.

B.10 THE ROLE OF DQIs IN THE PROJECT LIFECYCLE

DQIs play a number of important roles during the project planning phase such as calculating 

relevant indicators from historical data to support the design of new efforts, establishing MQOs,

identifying future DQI needs, and specifying quantitative and qualitative requirements. 

The role of DQIs during project planning focuses on assumptions (typically based on analysis of 

historical performance data) made during the development of DQOs and an associated statistical 
QAPP Handbook B-15 December 2012 Agency Review Draft

design of a project. For example, design optimization may involve an analysis of the major 

sources of sampling and measurement errors, as well as spatial and temporal variability that will 

contribute to uncertainty. Total study error impacts the ability to achieve the limits on decision 

error specified during Step 6 of the DQO process (EPA 2006a). Depending on the relative

contribution of different components of total study error (especially components of total study

precision), different choices may be made to cost-effectively achieve the specified DQOs. 

Project DQOs together with individual MQOs for precision, bias, and sensitivity are critical 

inputs to developing alternative sampling and analysis designs. Alternative designs should be 

developed that reflect different combinations of statistical sampling and measurement schemes. 

The expected performance of these designs may be calculated using these indicator 

DQOs/MQOs as inputs; the output of this effort is only as valid as the assumptions (including the 

quality and relevance of the indicators) that are made. In addition, once the critical DQIs are 

identified (those to which the final data quality is expected to be most sensitive), important 

design choices concerning QA/QC samples that enable estimates of the DQIs can be made. The 

purpose of such an analysis is to determine the likely output before the actual obligation of 

resources. Understanding the critical relationships between DQOs and MQOs is essential to the 

generation of data of known quality.

Decisions about what DQIs are appropriate in a study and what MQOs are to be achieved to 

meet the project DQOs should be documented in the QAPP. 

By conducting interim evaluations of the performance of the measurement system, project staff 

can identify whether to implement adjustments or corrective action to keep measurement systems 

in control before the project budget and schedule are expended. This is especially true for 

ongoing measurement programs such as monitoring efforts required for compliance programs, 

because mid-course corrections can be made based on feedback from the last sampling effort, or 

last series of samples. To support these QC measures, specific DQIs should be identified and 

data collected to support their calculation. In addition, MQOs assess and interpret the QC data 

results. Based on these types of comparisons, corrective actions can be taken as needed to 

improve project performance and increase the probability that data will be adequate to support 

the intended use. 

Too often, QC data are generated, but not evaluated at all, or not evaluated in a timely, 

meaningful way. The DQIs discussed in this document can be used not only in a real-time QC 

mode, but also in the assessment of the overall performance of a program over some period of 

time such as a field season. For example, indicators of precision, accuracy, and sensitivity can be 

evaluated over the field season to determine how well the program was able to measure critical 

variables. 

Following a data collection effort, determination of the adequacy of the data can be made via 

data verification, validation, and an evaluation of usability. The purpose of those data review 

steps is to determine whether the data set will support a decision with the desired degree of 

certainty. It is important to consider the performance and representativeness of the measurement 

effort prior to reaching conclusions regarding data adequacy. Step B of data validation (see 

Appendix E) focuses specifically on attainment of the MQOs set as quantitative goals for DQIs. 

Data usability then revolves around whether an adequate number of samples was obtained, given 

the observed measurement, spatial and temporal variability, and given the actual magnitude of 
QAPP Handbook B-16 December 2012 Agency Review Draft

the measurements made (relative to levels of concern) to meet the DQOs. If a data collection 

effort fails to generate adequate data to meet DQOs, then interest in DQIs is heightened, 

especially if there is a desire to “diagnose” which assumptions proved valid or invalid. Having 

documented the basis for the design based on DQIs, project staff can identify where the system 

failed to perform adequately and determine what additional data will be required to support the 

decision.
QAPP Handbook C-1 December 2012 Agency Review Draft

APPENDIX C

QUALITY CONTROL SAMPLES

Figures C-1 and C-2 depict a measurement process that involves several steps. Each step has the 

potential to introduce variability or bias that might influence the quality of project data. 

Estimates of the components of overall study variability and the relative contribution of 

measurement (within-unit) precision are important inputs to the statistical design process. The 

QAPP should describe the processes used to estimate and monitor the magnitude of at least the 

most important potential measurement error sources. Quality planning for measurements 

simultaneously establishes MQOs appropriate for the project and data use and defines the 

required DQIs. The data sources for DQIs are often derived from samples inserted in the 

sampling process by field or laboratory personnel on a frequency specified in the QAPP. Figures 

C-1 and C-2 suggest the types of QC samples that might be identified as the source of the 

underlying data for precision (Figure C-1) and bias (Figure C-2) DQI calculations.

Figure C-1. Total Sampling and Measurement Process Denoting the Use of QC Samples 

to Measure Components of Total Study Precision
QAPP Handbook C-2 December 2012 Agency Review Draft

Figure C-2. Total Sampling and Measurement Process Denoting the Use of QC Samples 

to Measure Components of Total Study Bias

Figures C-1 and C-2 are illustrative only with “statistical sampling” error (variance calculated 

from the sample analytical results) far exceeding all other errors (variance calculated from the 

QC sample analytical results). The results from routine QC samples often can be used to 

construct useful DQIs; however, some thought should be given to what support such data 

provide to the quality goals set out in the QAPP. One use of routine QC results that is generally

applicable is assessment of the laboratory's internal operations; for example, the routine analysis 

of calibration blanks to test for inorganic contamination or a problem in system calibration. In 

this case, a DQI could be just the actual blank result, expressed in appropriate units, with an 

MQO equal to the laboratory’s internal acceptance criterion. A poor result from a routine 

calibration blank should result in an immediate corrective action by the laboratory. This action 

may be limited to simply rerunning a calibration blank to demonstrate that the first result was an 

aberration. While it is important to verify the laboratory is monitoring and controlling the quality 

of its internal operations, results from samples like a calibration blank are less important in 

project planning from the DQO perspective than other potential sources of error that may not be 

controlled.
QAPP Handbook C-3 December 2012 Agency Review Draft

The same can be said for overall analytical calibrations. A properly calibrated analytical system 

is always required and expected for project samples analyses. Calibration data verify internal 

quality control, which is assumed to take place and is therefore not directly discussed during the 

DQO planning process. Instead, the quality attributes of greatest utility from a DQO planning 

and statistical design perspective include estimates of the overall (total) study variability and an 

understanding of the relative contribution of significant components of this total. The goal of 

analyzing components of bias is to reduce the most manageable components to minimal levels so 

the Total Study Error (as measured by the total sample variability) is reduced to a combination of 

random errors (the components of variability).

To establish the MQOs, the sensitivity of overall study error to the precision of various 

subcomponents of the total (e.g., the analytical component of measurement error) should be 

examined. A simplistic approach for examining the relative contribution of different components 

of total study variability begins from the assumption that total variability can be separated into 

individual, additive, components: 



 =  +  +  + ⋯

Using the framework for decomposition of errors shown in Figures C-1 and C-2 the sum of

errors in an environmental investigation might be 



 = 

 + 





 = 

 + 

 + 

where: T = total study error

w = within-sampling unit error

b = between-sampling unit error

s = small-scale error

m = measurement error

By identifying the largest of these components, decisions on method selection and laboratory

requirements (e.g. repeated measurements) can be made. Working with variances can be difficult 

conceptually because the terms are squared. A visualization tool using the more intuitive 

standard deviation statistic may be used for examining the additive relationship between 

precision components. The visualization tool takes advantage of Pythagoras' Theorem 

concerning the sides of a right triangle (the square of the hypotenuse is equal to the sum of the 

squares of the other two sides). Figure C-3 is a graphical representation of the most basic 

division of total study variability into the within-unit (measurement and small-scale) and 

between-unit (field or spatial) variability. Note that in this model the lengths of the sides of the 

triangle are directly proportional to the standard deviation, or precision, of the different 

components. The total standard deviation is obtained from the results of the actual randomly

located field samples while the estimate of within-unit precision is best estimated using 

collocated samples. The between-unit variability is inferred from the other two sides of the 

triangle. The dimensions in Figure C-3 represent the relative scales often encountered in 

environmental sampling, that is, between-unit variability generally dominates total study 

variability.
QAPP Handbook C-4 December 2012 Agency Review Draft

The requirements for precision can be stated at the very highest level (e.g., the total study error 

must not be greater than x%), or separated into between- and within-unit precision, or even 

further as suggested by Figure C-3. The QAPP should document quantitative MQOs for both 

between-unit and within-unit precision, and perhaps for more specific sources of imprecision that 

is needed to meet project-specific needs. In addition, the QAPP should document the corrective 

actions to be performed in the event any of the performance criteria are not met. 

Figure C-3. Components of Total Study

The QAPP should identify QC activities needed for each sampling, analysis, or measurement 

technique. For each required QC activity, the associated method or procedure, acceptance 

criteria, and corrective action should be listed. As standard methods are often vague or 

incomplete in specifying QC requirements, simply relying on the cited method to provide this 

information is usually insufficient. QC activities for the field and the laboratory include, but are 

not limited to, the use of blanks, duplicates, matrix spikes, laboratory control samples, 

surrogates, or second column confirmation. A list of various kinds of QC samples, as shown in 

Figures C-1 and C-2, are shown in Table C-1. 

Table C-1. Uses of Various QC Samples

QC Sample Description Purpose

Field 

or 

Lab

Bias or 

Precision

Field 

duplicate, 

collocated 

samples

Two or more independent samples 

collected from side-by-side 

locations at the same point in time 

and space

To assess precision of the total method, 

including sampling, analysis, and site 

heterogeneity

Field Precision

Field 

duplicate 

(split), 

subsamples

Duplicate samples resulting from 

one sample collection at one sample 

location (may be repeated collection 

or split of original sample)

To evaluate the effects of withinsample heterogeneity

Field Precision
QAPP Handbook C-5 December 2012 Agency Review Draft

QC Sample Description Purpose

Field 

or 

Lab

Bias or 

Precision

Matrix spike 

duplicate

A duplicate sample prepared 

simultaneously as a split of the 

matrix spike sample, each spiked 

with identical, known 

concentrations of targeted analytes

To determine the precision of the

laboratory analytical process for 

specific analytes in a sample matrix

Field Precision

Equipment 

blank

A clean water sample poured over 

or through decontaminated field 

sampling equipment 

To assess the adequacy of the 

decontamination process (also called 

rinse blank or rinsate blank)

Field Bias

Field blank A clean sample exposed to 

sampling conditions, transported to 

the laboratory, and treated as an 

environmental sample

Used to provide information about 

contaminants that may be introduced 

during sample collection, storage, and 

transport (may also serve as a rinsate 

blank)

Field Bias

Field matrix 

spike

A sample prepared by adding a 

known concentration of a target 

analyte to an aliquot of a specific 

homogenized environmental sample 

for which an independent estimate 

of the target analyte concentration is 

available

Accompanied by an independent 

analysis of the unspiked aliquot of the 

environmental sample, spiked samples 

are used to determine the potential bias 

introduced due to specific matrix 

effects

Field Bias

Trip blank A clean water sample transported 

from the sampling site to the 

laboratory for analysis without 

having been exposed to sampling 

procedures

To assess whether contamination is 

introduced during sample shipment 

(typically analyzed for volatile 

constituents)

Field Bias

Instrument 

replicate

Two or more analyses of the same 

sample designed to evaluate the 

precision of the analyses (also 

known as analytical replicates)

To evaluate the precision of the 

analyses at the instrument level only

Lab Precision

Laboratory 

duplicates/ 

replicates

Two or more representative portions 

taken from one homogeneous 

sample by the laboratory and 

analyzed in the same laboratory

To assess the overall laboratory 

preparatory and analytical precision

Lab Precision

Laboratory 

split sample

Two or more representative portions

of the same sample, analyzed by at 

least two different laboratories 

and/or methods. Prior to splitting, a 

sample is mixed (except volatiles, 

oil and grease, or when otherwise 

directed) to minimize sample 

heterogeneity

To assess precision, variability, and

data comparability between different 

laboratories

Lab Precision
QAPP Handbook C-6 December 2012 Agency Review Draft

QC Sample Description Purpose

Field 

or 

Lab

Bias or 

Precision

Instrument 

performance 

check sample

A sample of known composition 

analyzed concurrently with 

environmental samples to verify the 

performance of one or more 

components of the analytical 

measurement process (e.g., 

retention time, resolution, recovery, 

degradation, and calibration 

verification) 

To evaluate the potential bias 

introduced from the instrument (as part 

of the measurement) 

Lab Bias

Laboratory 

fortified blank

A low-level laboratory control 

sample (e.g., at the quantitation 

limit) (also known as laboratory 

spike or blank spike)

To evaluate laboratory preparatory and 

analytical sensitivity and bias for 

specific compounds

Lab Bias

Matrix spike A known concentration is injected 

into the matrix and then analyzed

To determine the recovery rate and 

therefore the bias

Lab Bias

Method blank 

(or extraction 

blank)

A sample of a matrix similar to the 

batch of associated samples in 

which no target analytes or 

interferences are present at 

concentrations that impact the 

analytical results, processed and 

analyzed simultaneously and under 

the same conditions as the samples

To evaluate the potential overall 

positive bias introduced by sample 

processing and measurement

Lab Bias

PT sample A sample of composition unknown 

to the laboratory or analyst, but 

known precisely by the PT sample 

vendor (also known as PE samples)

To assess potential bias and therefore 

capability to produce results within 

acceptable criteria PT samples can fall 

into three categories: (1) 

prequalification, conducted prior to a 

laboratory beginning project work, to 

establish initial proficiency; (2) 

periodic (e.g., quarterly, monthly, or 

episodic), to establish ongoing 

laboratory proficiency; and (3) batchspecific, which is conducted 

simultaneously with analysis of a 

sample batch. 

Lab Bias

Reagent blank An aliquot of clean water or solvent 

analyzed with the analytical batch 

and containing all the reagents in 

the same volume as used in the 

processing of the sample

To evaluate the potential bias 

introduced through reagents used 

during testing

Lab Bias

Note that although field-level QC will provide valuable information on total error it cannot 

identify the source of that error. It should be expected that the contributions to total error from 

the laboratory QC samples will be quite small but are useful in identifying where relatively large 

amounts of error are located.

Within the QAPP, appropriate QC activities should be identified (Figures C-1 and C-2 may be 

helpful in thinking about what types of QC activities fit project needs). 
QAPP Handbook C-7 December 2012 Agency Review Draft

The QAPP should state the frequency of analysis for each type of QC activity, and the spike 

compounds, sources, and levels. The QAPP should also state or reference the required control 

limits for each QC activity, and the corrective action required when control limits are exceeded,

and how the effectiveness of the corrective action shall be determined and documented.

The QAPP should describe, reference, or attach the procedures to be used to calculate applicable 

statistics (e.g., precision and bias). Copies of the formulae are acceptable as long as the 

accompanying narrative or explanation specifies clearly how the calculations will address 

potentially difficult situations such as missing data values, “less than” or “greater than” values, 

and other common data qualifiers.
QAPP Handbook D-1 December 2012 Agency Review Draft

APPENDIX D

DATA REVIEW, VERIFICATION, AND VALIDATION FOR PROJECT PURPOSES

Data review is the process in which data are examined and evaluated to varying levels of detail 

and specificity by a variety of personnel who have different responsibilities within the datamanagement process. The process includes verification, validation, and usability assessment. 

These QAPP elements encompass the data review activities used to ensure that only

scientifically valid data that are of known and documented quality and meet the DQOs (PQOs

depending on the organization) are used in making environmental decisions. The approach used 

for data review of a project must be appropriate to project requirements.

Although data review takes place after the data have been generated, determination of the type of 

data review that is required to meet DQOs begins during the planning phase of the project. Key

questions regarding data review that must be answered during the project planning stage include

(but are not limited to):

 What Measurement Quality Objectives (MQOs) are necessary to achieve the required 

DQOs such as precision, accuracy, representativeness, comparability, sensitivity, and 

completeness? (see also EPA 2006a)

 What data review inputs, activities, and outputs will be required for this project?

 What entities will be responsible for each step of the data review process and what are 

their relationships to those responsible for the data-production process?

 How will the implementation of the data review process and its results integrate with the 

overall project decision timeline? and

 What is the extent of data review and the availability and appropriate use of streamlining 

tools?

There are three distinct data review steps that are used to ensure that project data quality needs

are met. These data review steps are required for all data collected and used in environmental 

projects.

Data verification entails confirmation by examination and provision of objective evidence that 

the validated information fulfils specified requirements (sampling and analytical), or 

requirements mandated by a contract or agreement.

Data validation means confirmation by examination and provision of objective evidence that the 

particular requirement for which the data or information was collected is fulfilled. It includes the 

process of checking whether the information meets the project’s specifications.

Verification and validation extends to evaluating against criteria based on the quality objectives 

developed in the QAPP (e.g., the QAPP and MQOs). The purpose of validation is to assess the 

performance of the sampling and analysis processes to determine the quality of the specified 

data. It is divided into two subparts:

 assess and document compliance with methods, procedures, and contracts; and
QAPP Handbook D-2 December 2012 Agency Review Draft

 assess and document an evaluation of the ability of the data to meet the MQOs in the 

QAPP.

Table D-1 describes the objectives, scope, steps, and output of data review associated with each 

process term. The table identifies where the scope of the terms used or the steps involved in the 

process are expansions of current practice.

Table D-1. Three Steps of the Data Review Process

Step Objective Scope Activity

Verification Review to see if all the 

expected data are present

Sampling

Analysis

Completeness check

Validation Assess and document the 

performance of the fieldsample collection process

Assess and document the 

performance of the 

analytical process

Sampling

Analysis

Step A. Check compliance with 

method, procedure, and contract 

requirements

Step B. Compare with MQOs from the 

QAPP

Usability 

Evaluation

Assess and document 

usability to meet PQOs

Sampling

Analysis

Evaluate usability of data against 

DQOs and the decision to be made

Each step of the process is critical to the overall assessment of data quality and each step builds 

on the outcome of the previous step. The level of data review (types and amount of data 

reviewed) should be appropriate to the PQOs. Streamlining data review (validation in particular) 

is an option to consider that can potentially bypass some validation requirements, if allowed by 

the project’s data quality needs.

To perform the data review steps described above, reported analytical data must be supported by

complete data packages, as defined in the QAPP. Data packages include sample receipt and 

tracking information, chain-of-custody records, tabulated data summary forms, and raw 

analytical data for all field samples, standards, QC samples, and all other project-specific 

documents that are generated. If relevant raw data or sample information are not available or 

adequate to document data quality, then data review cannot be performed, and re-sampling or reanalysis must be considered. Existing data and model inputs/outputs should also be evaluated 

during data review.

This appendix describes what data review information should be included in the QAPP for each 

of the three data review steps: verification, validation (Steps A and B), and evaluation of 

usability. Table D-2 is provided as an example of inputs for data review and identifies the data 

review process step to which each input applies. These are only examples and are not intended to 

be either a minimum or comprehensive list of inputs.

Items 1 through 12 are to be found in the planning documents, 13 through 35 in the analytical 

data requirements document, 36 through 55 in the sampling documents. 
QAPP Handbook D-3 December 2012 Agency Review Draft

Table D-2. Example Inputs to the Data Review Process

Item

Verification Validation 

Step A

Compliance

Validation 

Step B

Comparison

1 Evidence of required approval of plan 

(QAPP)

X

2 Identification of personnel (those involved 

in the project and those conducting 

verification steps)

X

3 Laboratory name X

4 Methods (sampling and analysis) X X

5 Performance requirements (including QC 

criteria) for all inputs

X X X

6 Project quality objectives X X

7 Reporting forms X X

8 Sampling plans, location, maps, latitude, 

longitude, grids, and sample ID numbers

X X

9 Site identification X

10 SOPs (sampling and analytical) X X

11 Staff training and certification X

12 List of project-specific analytes X X

13 Case narrative X X X

14 Internal laboratory chain of custody X X

15 Sample condition upon receipt and storage 

records

X X

16 Sample chronology (time of receipt, 

extraction, and analysis)

X X

17 Identification of QC samples (sampling or 

lab, temporal, and spatial)

X X

18 Associated (batch or periodic) PT sample 

results

X X X

19 Communication logs X X

20 Copies of laboratory notebook, records, prep 

sheets

X X

21 Corrective action reports X X

22 Definitions of laboratory qualifier flags X X X

23 Documentation of corrective action results X X X

24 Documentation of individual QC results 

(e.g., spike or duplicate)

X X X

25 Documentation of laboratory method 

deviations

X X X

26 Electronic data deliverables X X

27 Instrument calibration reports X X X

28 Laboratory name X X

29 Laboratory sample identification numbers X X
QAPP Handbook D-4 December 2012 Agency Review Draft

Item

Verification Validation 

Step A

Compliance

Validation 

Step B

Comparison

30 QC sample raw data X X X

31 QC summary report X X X

32 Raw data X X X

33 Reporting forms, completed with actual 

results

X X X

34 Signatures for laboratory sign-off (e.g., 

laboratory QA manager)

X X

35 Standards traceability records (to trace 

standard source from National Institute of 

Standards and Technology, for example)

X X X

36 Chain of custody X X

37 Communication logs X X

38 Corrective action reports X X X

39 Documentation of corrective action results X X X

40 Documentation of deviation from methods X X X

41 Documentation of internal QA review X X X

42 Electronic data deliverables X X

43 Identification of QC samples X X X

44 Meteorological data from field (e.g., wind or 

temperature)

X X X

45 Sampling instrument decontamination 

records

X X

46 Sampling instrument calibration logs X X

47 Sampling location and plan X X

48 Sampling notes and drilling logs X X X

49 Sampling report (from field team to project 

manager describing sampling activities)

X X X

50 External assessment report X X X

51 External PT sample results X X

52 Laboratory assessment X X

53 Laboratory QA plan X X

54 Detection limit study information X X X

55 Lab accreditations X X

D.1 VERIFICATION

Verification is a completeness check that is performed before the data-review process continues 

in order to determine whether the required information (the complete data package) is available 

for further review. It involves a review of all data inputs to ensure that they are present. Table 

D-2 provides examples of the inputs for conducting the completeness check. 
QAPP Handbook D-5 December 2012 Agency Review Draft

Although this step is not designed for use in qualitative review (e.g., a compliance check that 

takes place during step B of the validation process), it is essential for ensuring the availability of 

sufficient information for subsequent steps of the data review process.

The planning process should establish verification procedures, which should be documented in 

the QAPP to ensure that data are evaluated properly, completely, and consistently for use in 

meeting DQOs. The procedures should address the following:

 the process that will be used to verify sample collection, handling, field analysis, and 

analytical laboratory project data; and

 the procedures and criteria that will be used to verify data information operations. (These 

operations include, but are not limited to, the electronic and/or manual transfer, entry, 

use, and reporting of data for computer models, algorithms, and databases; correlation 

studies between variables; and data plotting.)

Verification inputs may include items such as those listed in Table D-2. The description should 

detail how each item will be verified, when the activity will occur, and what documentation is 

necessary. Internal or external is in relation to the data generator. The resulting tables will 

describe the following:

 how sample collection, handling, and analysis procedures will be verified;

 how verification of field sampling, handling, and analysis activities will be documented 

(e.g., QC signatures in field logs and QC checklists);

 which sampling, handling, on-site analytical, and off-site laboratory data will be verified 

internally at the data-generator level;

 the end product of laboratory verification (e.g., laboratory-qualified data); and

 which sampling, on-site analytical, and off-site laboratory data will be verified by entities 

external to the data generator.

D.2 VALIDATION

The QAPP planning process must establish validation procedures and criteria. Project-specific 

validation procedures are developed to identify and qualify data that do not meet measurement 

performance criteria. Validation procedures and criteria are documented in the QAPP to ensure 

that data are evaluated properly, completely, and consistently for use in meeting DQOs. 

Validation guidance and documents may be attached to or referenced in the QAPP. EPA has

issued Guidance for Labeling Externally Validated Laboratory Analytical 

Data for Superfund Use, OSWER No. 9200.1-85, EPA 540-R-08-005 (EPA 2009c), which 

provides a standardized system for publishing the outputs of data validation. This or any other 

reasonable methodology for validation may be used provided it is properly documented.
QAPP Handbook D-6 December 2012 Agency Review Draft

The discussion of validation in the QAPP should address the following: the process that will be 

used to validate sample collection, handling, field analysis, and analytical laboratory project 

data;

 the specific validation process that will be used for each analytical group, matrix, and 

concentration level; and

 the procedures and criteria used to validate data information operations.(These may

include, but are not limited to, the electronic or manual transfer, entry, use, and reporting 

of data for computer models, algorithms, and databases; correlation studies between 

variables; and data plotting.)

Validation inputs include items such as those listed in Table D-2. The description should detail 

how each item will be validated, when the activity will occur, and what documentation is 

necessary. The resulting tables will describe the following:

 how sample collection, handling, and analysis procedures will be validated against the 

measurement performance criteria;

 how validation of field sampling, handling, and analysis activities will be documented 

(e.g., QC signatures in field logs and QC checklist s);

 which sampling, on-site analytical, and off-site laboratory data will be validated;

 the evaluative procedures used in validation to assess overall measurement error 

associated with the project, including DQIs; and

 the individual, identified by title and organizational affiliation, who is ultimately

responsible for data validation. This is the person (lead chemist or project chemist) who 

will sign the project validation reports.

In addition, the QAPP should identify the matrices, analytical groups, and concentration levels 

that each entity performing validation will be responsible for, as well as the criteria that will be 

used to validate those data. 

D.2.1 Validation Activities – Step A: Compliance

The examples listed in Table D-3 are of specific activities that may occur during an 

environmental project under Step A of the validation process (compliance with methods, 

procedures, and contracts) for both sampling and analytical data. Although these activities are 

organized separately, they may be performed at the same time and/or by the same people as

verification and Step B of validation activities.

D.2.2 Validation Activities – Step B: Comparison

The examples listed in Table D-4 are of specific activities that may occur during an 

environmental project under Step B of the validation process (comparison with MPCs 

documented in the QAPP) for both sampling and analytical data. These activities require that the 

validators have a complete copy of the QAPP, and they often involve all or parts of the project 

team. Some of the activities listed for Step A have a QAPP-specific review element and are 

therefore also listed as activities under Step B.
QAPP Handbook D-7 December 2012 Agency Review Draft

Table D-3. Validation Step A: Compliance with Methods, Procedures, and Contracts

Activity

Data Deliverables and 

QAPP

Ensure that all required information on sampling and analysis was 

provided (including planning documents).

Analytes Ensure that required lists of analytes were reported as specified in 

governing documents (i.e., method, procedure, or contract)

Chain-of-Custody Examine the traceability of the data from time of sample collection 

until reporting of data. Examine chain-of-custody records against 

contract, method, or procedural requirements.

Holding Times Identify holding time criteria, and either confirm that they were 

met or document any deviations. Ensure that samples were 

analyzed within holding times specified in method, procedure, or 

contract requirements. If holding times were not met, confirm that 

deviations were documented, that appropriate notifications were 

made (consistent with procedural requirements), and that approval 

to proceed was received prior to analysis.

Sample Handling Ensure that required sample handling, receipt, and storage 

procedures were followed, and that deviations were documented.

Sampling Methods and 

Procedures

Establish that required sampling methods were used and that 

deviations were noted. Ensure that sampling procedures and field 

measurements met performance criteria and that deviations were 

documented.

Field Transcription Authenticate transcription accuracy of sampling data (i.e., from 

field notebook to reports).

Analytical Methods 

and Procedures

Establish that required analytical methods (off-site laboratory and 

on-site analytical) were used and that any deviations were noted. 

Ensure that the QC samples met performance criteria and that any 

deviations were documented.

Data Qualifiers Determine that the laboratory data qualifiers were defined and 

applied as specified in methods, procedures, or contracts.

Laboratory 

Transcription

Authenticate accuracy of the transcription of analytical data (i.e., 

laboratory notebook to reporting form, or instrument to Laboratory

Information Management Systems).

Proficiency Testing Confirm acceptance of PT sample results against performance 

requirements as specified in methods, procedures, or contracts.

Standards Determine that standards are traceable and meet contract, method, 

or procedural requirements.

Communication Establish that required communication procedures were followed 

by field or laboratory personnel.

Assessments Review field and laboratory assessment reports and accreditation 

and certification records for performance on specific methods.

Validation Report –

Step A

Summarize deviations from methods, procedures, or contracts. 

Include qualified data and explanation of all data qualifiers.
QAPP Handbook D-8 December 2012 Agency Review Draft

Table D-4. Validation Step B: Comparison with MQOs

Activity

Data Deliverables and 

QAPP

Ensure that the data report from data validation step A was 

provided.

Deviations Determine the impacts of any deviations from sampling or 

analytical methods and SOPs. For example, confirm that the 

methods given in the QAPP were used and, if they were not, 

determine if data still meet MQOs. Consider the effectiveness and 

appropriateness of any corrective action.

Sampling Plan Determine whether the sampling plan was executed as specified 

(i.e., the number, location and type of field samples were collected 

and analyzed as specified in the QAPP).

Sampling Procedures Evaluate whether sampling procedures were followed with respect 

to equipment and proper sampling support (e.g., techniques, 

equipment, decontamination, volume, temperature, and 

preservatives)

Collocated Field 

Duplicates

Compare results of collocated field duplicates with criteria 

established in the QAPP.

Project Quantification 

Limits

Determine that quantitation limits were achieved, as outlined in 

the QAPP and that the laboratory successfully analyzed a standard 

at the quantitation limit.

Confirmatory Analyses Evaluate agreement of laboratory results.

Performance Criteria Evaluate QC data against project-specific performance criteria in 

the QAPP.

Data Qualifiers Determine that the data qualifiers applied in Step A were those 

specified in the QAPP and that any deviations from specifications 

were justified.

Validation Report –

Step B

Summarize the outcome of data comparison to MQOs in the 

QAPP. Include qualified data and explanation of all data 

qualifiers.

Further information on some further aspects of verification and validation may be found in 

Guidance on Environmental Data Verification and Data Validation, QA/G-8 (EPA 2008c).
QAPP Handbook E-1 December 2012 Agency Review Draft

APPENDIX E

CROSSWALKS TO OTHER DOCUMENTS

E.1 CROSSWALK: STANDARDS CLAUSES 7.5 TO 7.10 TO HANDBOOK QAPP 

AND HANDBOOK QMP

This crosswalk is intended to assist those who are developing a QAPP by linking the Internal and 

External Standards to this document Handbook QAPP and to the Quality Management Plan 

Handbook.

 Standards Clause Guidance

7.1 Management Representative for Quality QMP Handbook

7.2 Quality Management Plan QMP Handbook

7.3 Management Review and Reporting QMP Handbook

7.4 Personnel Competence QMP Handbook

7.5 Quality Assurance Project Plan QAPP Handbook : 1.7

7.6 Use of Information Technology Methods QAPP Handbook: 4.2

7.7 Systematic Planning QAPP Handbook: 2.2, 2.3, 3.2, 3.3, 4.2

7.8 Assessment of Data And Information QAPP Handbook: 2.4, 3.4, 4.4

7.9 Data Review and Usability Reporting QAPP Handbook: 2.5, 3.5, 4.5

7.10 Documents and Records Management QAPP Handbook: 2.2, 3.2, 4.2

7.11 Management of Quality Requirements for 

 External Agreements

QMP Handbook
QAPP Handbook E-2 December 2012 Agency Review Draft

E.2 CROSSWALK: STANDARDS ANNEX B TO QAPP HANDBOOK 

This crosswalk is intended to assist those who are developing a QAPP by linking the Standards

Annex B to this document QAPP Handbook.

 Standards Annex B QAPP Handbook Section

 

 B1

 

Introduction

1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, 

EPA POLICY 2106 and ANSI/ASQ E4-2004

1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN?

1.10 SUPERSESSION

B2 QAPP Responsibilities and 

Application

1 CHAPTER 1: INTRODUCTION

1.3 THE GRADED APPROACH

1.4 GENERIC QAPPs

1.5 PHASED QAPPs

1.9 PERIOD OF APPLICABILITY

B2.1 QAPP Preparation 

Responsibilities and Approvals

1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN?

1.6 WHEN SHOULD A QMP BE COMBINED WITH A 

QAPP?

1.7 DEVELOPING, REVIEWING, AND APPROVING A 

QAPP

B2.2 QA Implementation and

Revision

1.3 THE GRADED APPROACH

1.4 GENERIC QAPPs

1.5 PHASED QAPPs

1.8 DISTRIBUTING, IMPLEMENTING, AND MODIFYING 

A QAPP

B2.3 Applicability of QAPPs 1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, 

EPA POLICY 2106 and ANSI/ASQ E4-2004

1.9 PERIOD OF APPLICABILITY

B3.1 Overview and General 

Requirements

2.1 OVERVIEW OF QAPP ELEMENTS FOR THE 

COLLECTION OF DATA BY DIRECT 

MEASUREMENT

3.1 OVERVIEW OF QAPP ELEMENTS FOR 

EVALUATING EXISTING DATA

4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS
QAPP Handbook E-3 December 2012 Agency Review Draft

 Standards Annex B Handbook Section

B3.2 General Content 

Requirements

2.1 OVERVIEW OF QAPP ELEMENTS FOR THE 

COLLECTION OF DATA BY DIRECT 

MEASUREMENT

3.1 OVERVIEW OF QAPP ELEMENTS FOR 

EVALUATING EXISTING DATA

4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS

B3.3 Project Management 2.2 PROJECT MANAGEMENT: DIRECT MEASUREMENT

3.2 PROJECT MANAGEMENT: EXISTING DATA

4.2 PROJECT MANAGEMENT: MODELS

B3.4 Data Acquisition 2.3 DATA ACQUISITION: DIRECT MEASUREMENT

3.3 DATA ACQUISITION: EXISTING DATA

4.3 DATA ACQUISITION: MODEL DEVELOPMENT, 

MODIFICATION, AND USE 

B3.5 Assessments 2.4 ASSESSMENT: DIRECT MEASUREMENT

3.4 ASSESSMENT: EXISTING DATA

4.4 ASSESSMENT: MODEL ASSESSMENT ACTIONS

B3.6 Review, Evaluation of 

Usability, and Reporting 

Requirements

2.5 REVIEW, EVALUATION OF USABILITY, AND 

REPORTING REQUIREMENTS 

3.5 REVIEW, EVALUATION OF USABILITY, AND 

REPORTING REQUIREMENTS 

4.5 REVIEW, EVALUATION OF USABILITY: MODEL 

USABILITY AND REPORTING REQUIREMENTS
QAPP Handbook E-4 December 2012 Agency Review Draft

E.3 CROSSWALK: EPA QA/G-5 TO QAPP HANDBOOK

This crosswalk is provided for those who were familiar with the QA/G-5 guidance document. It 

maps the elements of the obsolete Guidance for Quality Assurance Project Plans, EPA QA/G-5,

to this QAPP Handbook.

QA/G-5 QAPP Handbook

1 INTRODUCTION

1.1 AN OVERVIEW OF QUALITY ASSURANCE (QA) PROJECT 

PLANS 1.6

1.2 EPA POLICY 1.1

1.3 CONTENT OF A QA PROJECT PLAN 1.2

1.4 QA PROJECT PLANS AND THE EPA QUALITY SYSTEM 1.1

1.5 DEVELOPING, REVIEWING, AND APPROVING A QAPP 1.7

1.6 DISTRIBUTING THE QA PROJECT PLAN 1.8

1.7 IMPLEMENTING THE QA PROJECT PLAN 1.8

1.8 RESOURCES Embedded in each Chapter

2 QA PROJECT PLAN ELEMENTS Chapter 2

2.1 Group A: PROJECT MANAGEMENT 2.2

2.1.1 Table of Contents 2.2.1

2.1.2 Distribution List 2.2.2

2.1.3 Distribution List 2.2.3

2.1.4 Project/Rask Organization 2.2.4

2.1.5 Problem Definition/Background 2.2.5

2.1.6 Project/Task Description 2.2.5

2.1.7 Quality Objectives and Criteria for Measurement Data 2.2.6

2.1.8 Special Training Needs/Certification 2.2.7

2.1.9 Documents and Records 2.2.8

2.2 Group B; DATA GENERATION AND ACQUISITION 2.3

2.2.1 Sampling Process Design (Experimental Design) 2.3.1

2.2.2 Sampling Methods 2.3.2

2.2.3 Sample Handling and Custody 2.3.3

2.2.4 Analytical Methods 2.3.4

2.2.5 Quality Control 2.3.5

2.2.6 Instrument/Equipment Testing, Inspection, and Maintenance 2.3.6

2.2.7 Instrument/Equipment Calibration and Frequency 2.3.6

2.2.8 Non-direct Measurements Chapter 3

2.2.10 Data Management 2.3.7

2.3 Group C: ASSESSMENT AND OVERSIGHT 2.4

2.3.1 Assessments and Response Actions 2.4.1

2.3.2 Reports to Management 2.5.5

2.4.3 Surveillance of Operations 2.5.4

2.4.4 Audits of Data Quality 2.4.4

2.4 Group D: DATA VALIDATION AND USABILITY 2.5

2.4.1 Data Review, Verification and Validation 2.5.1

2.4.2 Verification and Validation Methods 2.5.1

2.4.3 Reconciliation with User Requirements 2.5.4

3 PROJECTS USING EXISTING DATA Chapter 3

3.1 WHEN EXISTING DATA ARE USED ON AN 

ENVIRONMENTAL PROJECT 3.1

3.1.1 Determine Your Data Needs 3.2.5, 3.3.1

3.1.2 Identify Existing Data Sources That Might Meet Project Needs 3.3.1
QAPP Handbook E-5 December 2012 Agency Review Draft

QA/G-5 QAPP Handbook

3.1.3 Evaluate Existing Data Relative to Your Project’s Data Quality 

Specifications 3.4.1, 3.4.2

3.1.4 Document Quality Issues in Planning Documents or the Final 

Report 3.5.1. 3.5.3

3.2 ISSUES ON PREPARING A QA PROJECT PLAN FOR 

PROJECTS USING EXISTING DATA 3.5.2, 3.5.4

App A BIBLIOGRAPHY

Standards 11: References

Standards Annex. A: References

Standards Annex. B: References

Standards Annex. C: References

App B GLOSSARY OF QUALITY ASSURANCE AND RELATED 

TERMS Standards 9: Definitions

App C CHECKLIST USEFUL IN QA PROJECT PLAN REVIEW Appendix A
QAPP Handbook E-6 December 2012 Agency Review Draft

E.4 CROSSWALK: QAPP HANDBOOK TO EPA QA/G-5, QA/G-5M, AND QA/G-8

This crosswalk is provided for those who were familiar with the QA/G-5 guidance document. It 

maps the elements of this Handbook (CIO-G-05 QAPP) to the obsolete Guidance for Quality

Assurance Project Plans (QA/G-5), Guidance for Quality Assurance Project Plans for Modeling

(QA/G-5M), and Guidance on Environmental Data Verification and Data Validation (QA/G-8).

QAPP Handbook Section QA/G-5 Section

1 INTRODUCTION Chapter 1

1.1 QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 

2106 and ANSI/ASQ E4-2004 1.2, 1.4

1.2 WHAT IS A QUALITY ASSURANCE PROJECT PLAN? 1.1, 1.3

1.3 THE GRADED APPROACH 1.2

1.4 GENERIC QAPPs -

1.5 PHASED QAPPs -

1.6 WHEN SHOULD A QMP BE COMBINED WITH A QAPP? 1.1

1.7 DEVELOPING, REVIEWING, AND APPROVING A QAPP 1.5

1.8 DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP 1.6,1.7

1.9 PERIOD OF APPLICABILITY -

1.10 SUPERSESSION -

2

QAPP ELEMENTS FOR THE COLLECTION OF DATA BY 

DIRECT MEASUREMENT Chapter 2

2.1 OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF 

DATA BY DIRECT MEASUREMENT 2

2.2 PROJECT MANAGEMENT (PLAN) 2.1 Group A

2.2.1 Title, Version, and Approval/Sign-Off 2.1.1

2.2.2 Document Format and Table of Contents 2.1.2

2.2.3 Distribution List 2.1.3

2.2.4 Project Organization and Schedule 2.1.4

2.2.5 Project Background, Overview, and Intended Use of Data 2.1.5

2.1.6

2.2.6 Data/Project Quality Objectives and Measurement Performance 

Criteria 2.1.7

2.2.7 Special Training Requirements and Certification 2.1.8

2.2.8 Documentation and Records Requirements 2.1.9

2.3 DATA ACQUISITION (DO) 2.2 Group B

2.3.1 Data Collection Procedure, Experimental Design, and Sampling Tasks 2.2.1

2.3.2 Sampling Procedures and Requirements 2.2.2

2.3.3 Sample Handling, Custody Procedures, and Documentation 2.2.3

2.3.4 Analytical Methods Requirements and Task Design 2.2.4

2.3.5 Quality Control Requirements 2.2.5

2.3.6 Instrument/Equipment Testing, Calibration and Maintenance 

Requirements, Supplies and Consumables 2.2.6, 2.2.7

2.3.7 Data Management Requirements 2.2.10

2.4 ASSESSMENTS (CHECK) 2.3 Group C

2.4.1 Technical Systems Assessments 2.3.1

2.4.2 Performance Audits of Measurement and Analytical Systems 2.3.1

2.4.3 Surveillance of Operations 2.5.4

2.4.4 Audits of Data Quality 2.4.4

2.4.5 Qualitative and Quantitative Comparisons to Acceptance Criteria -

2.4.6 Interim Assessments of Data Quality -

2.4.7 Evaluation of Unconventional Measurements -

2.4.8 Evaluation of Unconventional Monitoring Projects -
QAPP Handbook E-7 December 2012 Agency Review Draft

QAPP Handbook Section QA/G-5 Section

2.5 REVIEW, EVALUATION OF USABILITY, AND REPORTING 

REQUIREMENTS (ACT)

2.4 Group D

2.5.1 Data Verification and Validation Targets and Methods 2.4.1, 2.4.2

2.5.2 Quantitative and Qualitative Evaluations of Usability -

2.5.3 Potential Limitations on Data Interpretation -

2.5.4 Reconciliation with Project Requirements 2.4.3

2.5.5 Reports to Management 2.3.2

3 QAPP ELEMENTS FOR EVALUATING EXISTING DATA Chapter 3

3.1 OVERVIEW OF QAPP ELEMENTS FOR EVALUATING 

EXISTING DATA

2.2.9

3.2

3.2 PROJECT MANAGEMENT (PLAN) 2.1 Group A

3.2.1 Title, Version, and Approval/Sign-Off 2.1.1

3.2.2 Document Format and Table of Contents 2.1.2

3.2.3 Distribution List 2.1.3

3.2.4 Project Organization and Schedule 2.1.4

3.2.5 Project Background, Overview, and Intended Use of Data 2.1.5, 2.1.6

3.2.6 Data/Project Quality Objectives and Measurement Performance 

Criteria 3.1.3

3.2.7 Special Training Requirements and Certification 2.1.8

3.2.8 Documentation and Records Requirements 3.1.4

3.3 DATA ACQUISITION (DO) 2.2 Group B

3.3.1 Proposed Data Source Originator and Publication Information 3.1.2

3.3.2 Data Format and Accessibility 2.2.9

3.3.3 Establishment of Acceptance Criteria 3.1.1

3.3.4 Sample Data Collection Methodology -

3.3.5 Quality Program and Quality Assurance Procedures Used by Data 

Originator -

3.3.6 Documentation of Sample Quality Assurance Procedures -

3.3.6 Data Management Requirements 2.2.10

3.4 ASSESSMENTS (CHECK) 2.3 Group C

3.4.1 Qualitative Comparisons to Acceptance Criteria 3.1.3

3.4.2 Quantitative Comparisons to Acceptance Criteria 3.1.3

3.4.3 Assessments to Other Criteria -

3.4.4 Interim Assessments of Data Quality -

3.4.5 Evaluation of Unconventional Measurements -

3.4.6 Evaluation of Unconventional Monitoring Projects -

3.5 REVIEW, EVALUATION OF DATA USABILITY, AND 

REPORTING REQUIREMENTS (ACT)

2.4 Group D

3.5.1 Data Verification and Validation Targets and Methods 2.4.1, 2.4.2

3.5.2 Quantitative and Qualitative Evaluations of Usability 3.1.4

3.5.3 Potential Limitations on Data Interpretation 3.2

3.5.4 Reconciliation with Project Requirements 2.4.3

3.5.5 Reports to Management 3.1.4

4

QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, 

AND USE OF MODELS EPA QA/G-5M

4.1 OVERVIEW OF QAPP ELEMENTS FOR MODELS G-5M Chapter 4

4.2 PROJECT MANAGEMENT (PLAN) G-5M 4.1

4.2.1 Title, Version, and Approval/Sign-Off G-5M 4.1.1

4.2.2 Document Format and Table of Contents G-5M 4.1.2

4.2.3 Distribution List G-5M 4.1.3

4.2.4 Project Organization and Schedule G-5M 4.1.4

4.2.5 Project Background, Overview, and Intended Use of the Model G-5M 4.1.5
QAPP Handbook E-8 December 2012 Agency Review Draft

QAPP Handbook Section QA/G-5 Section

4.2.6 Data/Project Quality Objectives and Measurement Performance 

Criteria G-5M 4.1.6

4.2.7 Special Training Requirements and Certification G-5M 4.1.8

4.2.8 Documentation and Records Requirements G-5M 4.1.9

4.3 DATA AQCQUISITION: MODEL DEVELOPMENT, 

MODIFICATION, AND USE (DO) G-5M 4.2

4.3.1 Problem Specification and Identification of Model Purpose and Scope G-5M 3.3

4.3.2 Model Development or Selection Process G-5M 4.2.1

4.3.3 Data Requirements for Model Input G-5M 4.4.2

4.3.4 Evaluation of the Model G-5M 4.4.1

4.4 ASSESSMENTS: MODEL ASSESSMENT ACTIONS (CHECK) G-5M 4.3

4.4.1 Assessments to Acceptance Criteria and Responses/Corrective Actions G-5M 4.3.1

4.4.2 Data Management Tasks G-5M 4.3.2

4.4.3 Model Output Sensitivity and Uncertainty Analysis G-5M 4.4.1

4.5 REVIEW, EVALUATION OF USABILITY: MODEL USABILITY 

AND REPORTING REQUIREMENTS (ACT) G-5M 4.4

4.5.1 Model Evaluation Methods and Activities G-5M 4.4.1, 4.4.2, 4.4.3

4.5.2 Description of Model Documentation G-5M 4.1.9

4.5.3 Specifications for Model Maintenance and User Support G-5M 4.2.3

4.5.4 Reports to Management G-5M 4.3.2

App A CHECKLIST OF QAPP ELEMENTS Appendix C

App B DATA QUALITY INDICATORS -

B.1 Sources of Measurement Uncertainty -

B.2 Establishing MQOs in the Context of DQOs -

B.3 Precision -

B.4 Bias -

B.5 Accuracy -

B.6 Representativeness -

B.7 Comparability -

B.8 Completeness -

B.9 Sensitivity -

B.10 The Role of DQIs in the Project Lifecycle -

App C QUALITY CONTROL SAMPLES -

App D DATA REVIEW, VERIFICATION, AND VALIDATION FOR 

PROJECT PURPOSES EPA QA/G-8

D.1 Verification G-8 2.1, 2.2, 2.3, 2.4

D.2 Validation G-8 3.1, 3.2, 3.3, 3.4

D.2.1 Validation Activities – Step A: Compliance G-8 5.2

D.2.2 Validation Activities – Step B: Comparison G-8 5.2

App E CROSSWALKS -
QAPP Handbook E-9 December 2012 Agency Review Draft

E.5 CROSSWALK: UFP-QAPP WORKBOOK TO QAPP HANDBOOK

This crosswalk is intended to assist those who are developing a QAPP using the UFP-QAPP 

worksheet format. The sections of this document that provide help for each worksheet of the 

UFP-QAPP are indicated below.

 UFP-QAPP Worksheet QAPP Handbook Section

1 & 2 Title and Approval Page 2.2.1 Title, Version, and Approval/Sign-Off

3 & 5 Project Organization and QAPP Distribution 2.2.3 Distribution List

2.2.4 Project Organization and Schedule

4 , 7 & 8 Personnel Qualifications and Sign-off Sheet 2.2.1 Title, Version, and Approval/Sign-Off

2.2.7 Special Training Requirements and 

Certification

6 Communication Pathways 2.2.4 Project Organization and Schedule

9 Project Planning Session Summary 2.2.5 Project Background, Overview, and Intended 

Use of Data

10 Conceptual Site Model 2.2.5 Project Background, Overview, and Intended 

Use of Data

11 Project/Data Quality Objectives 2.2.6 Data/Project Quality Objectives and 

Measurement Performance Criteria

12 Measurement Performance Criteria 2.2.6 Data/Project Quality Objectives and 

Measurement Performance Criteria

13 Secondary Data Uses and Limitations Chapter

3

QAPP ELEMENTS FOR EVALUATING 

EXISTING DATA

14 & 16 Project Tasks & Schedule 2.2.4 Project Organization and Schedule

15 Project Action Limits and LaboratorySpecific Detection / Quantitation Limits2.2.6 Data/Project Quality Objectives and 

Measurement Performance Criteria

17 Sampling Design and Rationale 2.3.1 Sample Collection Procedure, Experimental 

Design, and Sampling Tasks

18 Sampling Locations and Methods 2.3.1 Sample Collection Procedure, Experimental 

Design, and Sampling Tasks

2.3.2 Sampling Procedures and Requirements

19 & 30 Sample Containers, Preservation, and Hold 

Times

2.3.2 Sampling Procedures and Requirements

20 Field QC Summary 2.3.5 Quality Control Requirements

21 Field SOPs 2.3.2 Sampling Procedures and Requirements

22 Field Equipment Calibration, Maintenance, 

Testing, and Inspection

2.3.6 Instrument/Equipment Testing, Calibration 

and Maintenance Requirements, Supplies and 

Consumables

23 Analytical SOPs 2.3.4 Analytical Methods Requirements and Task 

Description

24 Analytical Instrument Calibration 2.3.6 Instrument/Equipment Testing, Calibration 

and Maintenance Requirements, Supplies and 

Consumables
QAPP Handbook E-10 December 2012 Agency Review Draft

 UFP-QAPP Worksheet QAPP Handbook Section

25 Analytical Instrument and Equipment 

Maintenance, Testing, and Inspection 

2.3.6 Instrument/Equipment Testing, Calibration 

and Maintenance Requirements, Supplies and 

Consumables

26 & 27 Sample Handling, Custody, and Disposal 2.3.3 Sample Handling, Custody Procedures, and 

Documentation

28 Analytical Quality Control and Corrective 

Action

2.3.5 Quality Control Requirements

29 Project Documents and Records 2.2.8 Documentation and Records Requirements

31, 32 &

33

Assessments and Corrective Action 2.4 ASSESSMENTS AND DATA REVIEW 

(CHECK)

2.5.5 Reports to Management

34 Data Verification and Validation Inputs 2.5.1 Data Verification and Validation Targets and 

Methods

35 Data Verification Procedures 2.5.1 Data Verification and Validation Targets and 

Methods

36 Data Validation Procedures 2.5.1 Data Verification and Validation Targets and 

Methods

37 Data Usability Assessment 2.5.2 Quantitative and Qualitative Evaluations of 

Usability

2.5.3 Potential Limitations on Data Interpretation

2.5.4 Reconciliation with Project Requirements