Quality Assurance Project Plan

For the Economic Portion of the Coalbed 

Methane Detailed Questionnaires and 

Coalbed Methane Economic Analysis Task

(Element 1.A)

Prepared for:

U.S. Environmental Protection Agency

Engineering and Analysis Division

Office of Water

1200 Pennsylvania Avenue, NW

Washington, D.C. 20460

Prepared by:

Eastern Research Group, Inc.

110 Hartwell Avenue

Lexington, MA 02421

March 15, 2012

EPA Contract No. 68-C-02-095

Work Assignment 9-37

QAPP—CBM Detailed Questionnaire and Economic Analysis 

ii

History of the QAPP for the Economics Portion of the Coalbed Methane Project

Rev 0

Delivery Date November 2, 2010

Work Assignment 8-05

Scope CBM Detailed Questionnaire 

(Part B) 

Comments Received NA

Comments 

Addressed

NA

Approval November 3, 2010 

Rev 1

Delivery Date January 4, 2012

Work Assignment 9-37

Scope CBM Detailed Questionnaire 

(Part B) and Economic 

Analysis Task 

Comments Received February 27, 2012

Comments 

Addressed

March 15, 2012

Approval
QAPP—CBM Detailed Questionnaire and Economic Analysis 

iii

Summary of Changes Made in Rev. 1

Applicable Section Change

QAPP Title Changed to reflect the broader scope of the QAPP

Entire QAPP Minor editorial changes and reorganization for clarity and 

readability and updates to Work Assignment number and revision 

number 

Approvals Updated to reflect ERG and EPA personnel changes on project

Distribution Updated to reflect ERG and EPA personnel changes on project

1. Introduction and 

Background

Updated to reflect new ERG WAM, a change in Engineering 

Contractor, expansion of scope, and extension of project history

2.1 Project/Task 

Organization

Updated Project Organization figure to reflect change in Engineering 

Contractor and updated text to reflect new staffing and lines of 

communication and reorganized section for clarity

2.2 Problem Definition 

and Background

Revised to reflect expanded scope and new tasks under the 

Economic Analysis Task

2.3 Project/Task 

Description

Revised to reflect expanded scope and new tasks under the 

Economic Analysis Task, change in Engineering Contractor 

3.4 Data Handling and 

Custody

Revised to reflect change in Engineering Contractor

3.5.2 Questionnaire 

Response Review and 

Followup

ERG deleted a sentence to reflect the fact that EPA no longer 

believes a letter informing operators of changes made to their data 

during data cleaning is required and made minor edits to reflect a 

change in initial callback procedures necessitated by a change in task 

priorities under Work Assignment 8-05

3.5.3 Questionnaire 

Response Coding and 

Data Entry

Revised to reflect more extensive proofing used to QA data entry

4.1 Assessments and 

Response Actions

Revised to reflect expanded scope

4.2 Reports to 

Management

Revised for compatibility with overall Work Assignment 9-37 

QAPP

6. Addendum to the 

QAPP

Added detailed discussion of QA procedures associated with the 

expanded scope of the economic analysis activities, including the 

analysis of the detailed questionnaire data, replacement of missing 

data, methodology development, application of methodologies to 

data, and report writing tasks.

Appendix A Deleted contact list
QAPP—CBM Detailed Questionnaire

iv

CONTENTS

(Element A.2)

1. INTRODUCTION AND BACKGROUND ..................................................................................... 1

2. PROJECT MANAGEMENT ELEMENTS .................................................................................... 1

2.1 Project/Task Organization (Element A.4)............................................................... 1

2.2 Problem Definition and Background (Element A.5) .............................................. 3

2.3 Project/Task Description (Element A.6)................................................................. 4

2.3.1 Finalizing the Detailed Questionnaire ........................................................ 5

2.3.2 Providing Survey Assistance ...................................................................... 5

2.3.3 Receipt and Internal Tracking of Questionnaires ....................................... 5

2.3.4 Developing a Survey Database and Data Dictionary.................................. 6

2.3.5 Reviewing, Transcribing, Coding, and Entering Respondent Data 

into Survey Database .................................................................................. 6

2.3.6 Following Up on Incomplete Surveys ........................................................ 6

2.3.7 Tabulating Data........................................................................................... 7

2.4 Quality Objectives and Criteria (Element A.7)....................................................... 7

2.4.1 Completeness.............................................................................................. 7

2.4.2 Accuracy ..................................................................................................... 7

2.4.3 Reasonableness........................................................................................... 8

2.5 Special Training Requirements/Certification (Element A.8).................................. 8

2.6 Documentation and Records (Element A.9)........................................................... 8

3. DATA GENERATION AND ACQUISITION ELEMENTS .............................................................. 1

3.1 Sampling Process Design (Element B.1)................................................................ 1

3.2 Sampling Methods (Element B.2)........................................................................... 1

3.3 Completeness.......................................................................................................... 1

3.3.1 Accuracy and Reasonableness.................................................................... 2

3.4 Data Handling and Custody (Element B.3)............................................................. 3

3.5 Data Management (Element B.10).......................................................................... 3

3.5.1 Questionnaire Non-Response Follow Up ................................................... 3

3.5.2 Questionnaire Response Review and Follow Up........................................ 4

3.5.3 Questionnaire Response Coding and Data Entry........................................ 5

4. ASSESSMENT AND OVERSIGHT ELEMENTS........................................................................... 1

4.1 Assessments and Response Actions (Element C.1)................................................ 1

4.2 Reports to Management (Element C.2)................................................................... 2

5. RECONCILIATION WITH USER REQUIREMENTS..................................................................... 1

6. ADDENDUM COVERING THE DATA ANALYSIS AND USE OF EXISTING DATA IN THE 

COALBED METHANE ECONOMIC ANALYSIS......................................................................... 1

6.1 Introduction............................................................................................................. 1

6.2 Project Description and Objectives......................................................................... 2

6.3 Data Needs, Sources, and Quality Criteria ............................................................. 3

6.4 Data Acquisition and Review ................................................................................. 7
CONTENTS (Continued)

Page

v

6.4.1 Existing Data Tracking Database (Tracking Database) and 

Development of Detailed Data Tables........................................................ 8

6.4.2 Development of Data Subsets..................................................................... 9

6.5 Use of Existing Data and Quality Verification..................................................... 10

6.5.1 Develop Methodologies and Evaluate Profile Information (Tasks 

8.a, 8.c, 8.d, and 8.f).................................................................................. 10

6.5.2 Calculate Impacts...................................................................................... 12

6.5.3 Check Results of Impact Estimates........................................................... 13

6.6 Project Reports (Deliverables).............................................................................. 15

7. REFERENCES ........................................................................................................................ 1

Appendix A: INSTRUCTIONS FOR SURVEY REVIEW & CODING
QAPP—CBM Detailed Questionnaire

vi

LIST OF TABLES

Page

1 QAPP Distribution.................................................................................vii

6-1 Existing Data Collection Criteria................................................................6-6

6-2 Industry Profile and Updated Production and Financial Data Acceptance Criteria ........6-5

6-3 Criteria for Acceptance of Replacement Data for Missing or Erroneous Data ...............6-6

6-4 Impact Estimate Quality Specification ............................................................................6-7

6-5 Summary of Procedures and Results Quality Verification............................................ 6-14

6-6 Key Products.................................................................................................................. 6-16

LIST OF FIGURES

Page

2-1 Project Organization (Element A.4) ................................................................................ 2-2

3-1 CBM Detailed Questionnaire Contact Record................................................................. 3-6
QAPP—CBM Detailed Questionnaire

vii

DISTRIBUTION

(Element A.3)

This Quality Assurance Project Plan (QAPP) was distributed to staff of the U.S. 

Environmental Protection Agency and Eastern Research Group, Inc. (Table 1). A copy of the 

document was provided to all Eastern Research Group involved in the project, including those 

who joined the project after publication of the QAPP.

Table 1. QAPP Distribution

Name

Title Contact Information Mailing Address

U.S. Environmental Protection Agency

Meghan Hessenauer

Technical Project Officer

(202) 566-1040 (phone)

hessenauer.meghan@epa.gov

U.S. Environmental Protection Agency

Office of Science and Technology

1200 Pennsylvania Avenue, NW (4303T)

Washington, DC 20460

(202) 566-1053 (fax)

Marion Kelly

Quality Assurance Coordinator

(202) 566-1045 (phone)

kelly.marion@epa.gov

Marla Smith

Work Assignment Manager

(202) 566-1047 (phone)

smith.marla@epa.gov

Ahmar Siddiqui

Alternate Work Assignment 

Manager

(202) 566-1044 (phone)

siddiqui.ahmar@epa.gov

James Covington, III

Economics Task Manager

(202) 566-1034(phone) 

covington.james@epa.gov

Janet K. Goodwin

Technology & Statistics Branch 

Chief

(202) 566-1060 (phone)

goodwin.janet@epa.gov

Eastern Research Group, Inc

Deborah Bartram

ERG Program Manager

(703) 633-1669 (phone)

Deborah.Bartram@erg.com

Eastern Research Group, Inc.

14555 Avion Parkway 

Suite 200

Chantilly, VA 20151-1102

(703) 263-7280 (fax)

Mary Willett

ERG Corporate Quality Assurance 

Manager and Project QA 

Coordinator

(703) 633-1615 (phone)

Mary.Willett@erg.com

Lori Weiss

Work Assignment Manager

(703) 633-1658 (phone

Lori.Weiss@erg.com

Anne Jones

Economics Task Manager

(781) 674-7328 (phone)

Anne.Jones@erg.com

Eastern Research Group, Inc

110 Hartwell Avenue

Lexington, MA 02421-3136

(781) 674-2851 (fax)

Melanie Sands

Environmental Policy Analyst

(781)-674-7321 (phone)

Melanie.Sands@erg.com

Christiana Marsden

Economist

(781) 674-7331 (phone)

Christiana.Marsden@erg.com
Section No. 1

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 2

1. INTRODUCTION AND BACKGROUND

The U.S. Environmental Protection Agency (EPA), acting through its Office of Science 

and Technology (OST), Engineering and Analysis Division (EAD), plans to revise the effluent 

limitations guidelines and standards (ELGs) for the Oil and Gas Point Source Category to control 

pollutant discharges from coalbed methane (CBM) produced water. EPA’s decision to proceed 

with a rulemaking was announced in the final 2010 Effluent Guidelines Program Plan (76 FR 

66286). The decision was based, in part, upon data collected from an industry survey conducted 

in 2008. EPA is now planning to consider the economic data received in the questionnaires in its 

rulemaking for the CBM industry. 

EPA tasked Eastern Research Group, Inc. (ERG) with supporting the Agency in its 

Economic Analysis under Work Assignment 9-37 of Contract No. 68-C-02-095. Specifically, 

EPA tasked ERG with finalizing the development of the database for Part B (economic and 

financial data) of the detailed questionnaire that had been developed under earlier work 

assignments (Work Assignments 8-37, 8-05, 7-05, 6-05, and 5-05) under the same contract. EPA 

has also tasked ERG with using the detailed questionnaire data, along with additional 

information, to analyze the potential impacts of produced water management practices on 

operators of CBM projects, CBM projects, production, and the wider economy.

The detailed questionnaire contains both technical (Parts A and C) and economic (Part B) 

sections. Currently, support to EPA in finalizing and analyzing Parts A and C of the 

questionnaire under Task 9-37 is covered by the Quality Assurance Project Plan for the Review 

and Data Entry of the Engineering Portions of the Coalbed Methane Questionnaire (1), the 

Quality Assurance Project Plan for the Coalbed Methane Existing Data Collection, dated 

August 25, 2011 (2), and the Data Analysis Addendum to the Quality Assurance Project Plan for 

the Coalbed Methane Existing Data Collection, dated November 30, 2011 (3). This document 

presents ERG’s quality assurance procedures for the review of Part B data, creation of a database 

to house the responses, economic analysis, and incorporation of secondary data into the analyses. 

The new activities for Work Assignment 9-37 are discussed as an addendum in Section 6.0. 

Quality assurance procedures related to survey design and statistical analyses are provided in the 

QAPP prepared by Battelle under a separate contract (Quality Assurance Project Plan for WA 2-

11: Statistical Support for Coalbed Methane Detailed Study; Contract No. EP-C-05-030) (4). 

Procedures for maintaining the CBM public record are provided in the Coalbed Methane Project 

File Management Plan prepared under this contract (Contract No. 68-C-02-095, Work 

Assignment 8-37; see Coalbed Methane Project File Management Plan, May, 2011). The Project 

File Management Plan is Appendix A of the approved technical QAPP (Contract No. 68-C-02-

095, Work Assignment 8-37; see Quality Assurance Project Plan for the Review and Data Entry 

of the Engineering Portions of the Coalbed Methane Questionnaire, dated July 13, 2011 (1).

This QAPP is responsive to applicable elements in EPA Requirements for Quality 

Assurance Project Plans QA/R-5 (5). It is a project-specific plan that describes the procedures 

that staff supporting EPA on the CBM Economic Analysis use to plan, implement, and assess 

project quality. 
Section No. 1

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 2 of 2

ERG has organized this document according to the elements outlined in the abovementioned EPA guidance document. Group A elements are addressed from the title page through 

Section 2.6 of the document, while applicable Group B elements are addressed in Section 3.0. 

Group C elements are addressed in Section 4.0, and applicable Group D elements are addressed 

in Section 5.0. The addendum addressing the economic analysis and use of secondary data is 

provided in Section 6.0. References are provided in Section 7.0.
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 9

2. PROJECT MANAGEMENT ELEMENTS

In this section, ERG discusses its project management approach for the assignment. The 

discussion is organized according to the nine elements EPA identified for inclusion in this 

section:

A.1 Title and Approval Sheet

A.2 Table of Contents

A.3 Distribution List

A.4 Project/Task Organization

A.5 Problem Definition/Background

A.6 Project/Task Description

A.7 Quality Objectives and Criteria

A.8 Special Training/Certification

A.9 Documents and Records

Elements A.1 to A.3 were provided earlier in this document. The remaining elements are 

discussed below.

2.1 Project/Task Organization (Element A.4)

Figure 2-1 presents the project organization for the CBM economics tasks. The chart 

depicts the staff with program management and technical responsibilities and those with QA/QC 

roles. It shows the relationship and lines of communication among all project participants, 

including those from EPA, as users of the data.

Deborah Bartram serves as ERG Program Manager overseeing all tasks performed under 

Contract No. 68-C-02-095. Lori Weiss serves as the ERG Work Assignment Manager (WAM)

for 9-37, Coalbed Methane (CBM) Effluent Guidelines Regulatory Support, of which the CBM 

Economics Analysis Task is one subtask. Ms. Weiss will be the principal contact for the EPA 

WAM on project issues, deliverables, and schedules. Ms. Weiss will be responsible for:

• Ensuring that the quality of work, schedule, and budget meets the requirements of 

the EPA/EAD effluent guidelines program;

• Providing technical direction to ERG staff and managing the daily activities on 

the project;

• Obtaining appropriate technical review of all ERG deliverables and ensuring 

deliverables conform to EPA’s technical review requirements; and

• Keeping the ERG Project QA Coordinator and the ERG Program Manager 

advised of ay quality problems that arise.

Ms. Mary Willett, will serve as Project QA Coordinator, responsible for ensuring that the 

requirements of this QAPP are implemented and documented. Ms. Willett also serves as ERG’s 

overall Corporate QA Manager (QAM), and as such ensures that project-specific QA/QC 
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 2 of 9

programs are commensurate with project objectives and with ERG’s quality system. Ms. Willet 

has no other role on the CBM project and is independent from the CBM project team.

On this work assignment, Ms. Willett, following consultation with ERG’s Program 

Manager (PM), has the authority to suspend the performance of any activities determined to be 

deviating from the established quality management procedures, work plan, or project-specific 

QAPPs until appropriate corrective actions are instituted. She can, at her discretion, conduct an 

audit of ERG’s QA activities during the period of performance of the assignment.

Figure 2-1. Project Organization (Element A.4)

ERG Corporate

QA Manager

ERG Project

QA Coordinator

ERG Technical

Reviewers

To be assigned

ERG Program

Manager

ERG Work

Assignment

Manager

ERG

Task Lead

ERG

Staff

EPA Project Officer

EPA Work

Assignment 

Manager

 

EPA

QA Officer

Lines of Communication

Reporting and Authority
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 3 of 9

Under Ms. Weiss, Anne Jones, Senior Economist and ERG Vice President, manages the 

CBM Economics Task, providing support to James Covington, III, EPA Economics Task 

Manager for the Economic Analysis Task. Anne Jones is supported by other ERG staff as 

needed. The staff adhere to the QA guidelines outlined in this QAPP when providing support 

under the work assignment. 

The ERG WAM is responsible for reviewing all deliverables before transmittals to EPA. 

This review ensures a consistent level of quality for the deliverables. In addition, it ensures 

consistency in data selection, definitions, and other aspects between the technical and economic 

support delivered under the work assignment. In addition, the ERG project QA Coordinator, 

ERG WAM, and ERG Economics Task Manager appoint Technical Reviewers as needed during 

the conduct of this assignment. The Technical Reviewers evaluate the quality of work products 

as they are being developed and once they are completed. The reviewers do not have any 

involvement in the products or processes for which they were assigned QA/QC responsibilities. 

The Technical Reviewers could be internal or external to the ERG Team. They have economics 

or policy backgrounds, as well as experience in survey design, administration, and analysis. (For 

further discussion on technical review procedures, see Section 4.1 of this QAPP.)

2.2 Problem Definition and Background (Element A.5)

EPA developed and administered the CBM screener and detailed questionnaires during 

the Detailed Study to collect information to determine whether it would be appropriate to revise 

the effluent guidelines for the Oil and Gas Point Source Category to control pollutant discharges 

from CBM produced water. The data are now being used in economic analyses for rulemaking 

activities. The data collection for the detailed study was intended to be comprehensive so that an 

additional survey would not be needed if EPA were to go forward with regulating the industry. 

Thus, the screener and detailed questionnaire databases were designed to provide the level of 

detail and the data needed to support an ELG.

The purpose of the questionnaires was to gather industry information that would enable 

EPA to go forward with analyses necessary to support an ELG (in addition to providing 

information suitable for a detailed study report). The information gathered supports: 

• Developing a profile of the CBM industry and its produced water management 

practices;

• Determining the scope of the industry and any subcategories;

• Determining whether technically and economically feasible control technologies 

or management strategies exist to reduce or eliminate the discharge of pollutants; 

and

• Determining whether the current regulatory controls are adequate.

In addition to answering these questions, the questionnaires will allow EPA to address the 

key questions and project outcome described below. EPA employed a two-phase sample design. 

In the first phase, EPA collected information about CBM projects from a census of CBM 

projects with more than two wells. The responses to the screener questionnaire used in this 
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 4 of 9

census served as the basis for EPA to select a sample of CBM projects to receive the detailed 

questionnaire. (For further discussion on sample design, see Section 3.1 of this QAPP.)

The detailed questionnaire was designed to answer the following key questions: 

• What pollutants are being discharged by CBM operations?

• What are the existing produced water management alternatives?

• How do produced water management alternatives vary by basin?

• What are the operation and maintenance costs of the control technologies?

• For what produced water characteristic is each management alternative feasible?

• What are the expected incremental pollutant removals attributable to each 

management alternative?

• What are the firm-level baseline financial characteristics?

• What are the historic and current capital and operating costs of CBM projects 

specific to each basin?

• What are the names, locations, and production histories of CBM extraction 

projects? 

• What are the gas and produced water production rates over the life of each 

project? 

• What is the economic impact (e.g., production losses, firm failures, impacts on 

small business) of each produced water management alternative?

• What is the status of current National Pollutant Discharge Elimination System 

(NPDES) and other operating permits?

• What are the non-water-quality impacts of each management alternative?

In terms of the survey implementation, data collection, and database development, this 

QAPP specifically addresses implementation of Part B, Financial Information, of the detailed 

questionnaire. Other aspects of the use of survey and other data and the economic analysis are 

addressed in the addendum in Section 6.0.

2.3 Project/Task Description (Element A.6)

EPA received approval from the Office of Management and Budget (OMB) for an 

industrial survey (screener and detailed questionnaires) under the Paperwork Reduction Act on 

February 29, 2009. EPA published two Federal Register Notices announcing the data collection 

efforts (73 FR 4556, January 25, 2008; 73 FR 40575, July 15, 2008). ERG support of this survey

consisted of:

• Revising and finalizing the detailed questionnaire; 

• Responding to questions on Part B received on the CBM Helpline; 

• Receiving copies of the detailed questionnaire and tracking these copies; 

• Developing a database dictionary and a database to house questionnaire data; 

• Reviewing Part B of questionnaires to prepare them for data entry, including 

transcribing financial data for some respondents from their submitted financial 

reports; and
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 5 of 9

• Entering the data into the prepared database.

ERG has completed the steps above. A brief description of these tasks is provided in the 

subsections that follow. The QA/QC activities associated with the tasks are discussed in Section 

3.0 of this QAPP. 

Those tasks currently underway or to be performed include the following:

• Conducting follow up on incomplete or problematic detailed questionnaires, if so 

directed by EPA;

• Summarizing questionnaire data;

• Identifying outliers and missing data and developing approaches for estimating 

missing data;

• Continuing to clean questionnaire data as outliers are identified or due to follow 

up;

• Developing economic analysis methodologies and implementing the 

methodologies to calculate economic impacts;

• Using survey data and existing data to create an industry economic profile; and

• Developing draft and final documents supporting economic analyses.

A brief description of these tasks is provided in Section 6.0, with the exception of conducting 

follow up., which is described in Section 2.3.6.

2.3.1 Finalizing the Detailed Questionnaire

ERG assisted EPA in finalizing the detailed questionnaire in October 2009. There were 

no significant changes to the detailed questionnaire––only edits to improve the look and ease of 

use. Questions for the detailed questionnaire were designed to elicit information in the areas

outlined in Section 2.2. This task has been completed.

2.3.2 Providing Survey Assistance 

ERG assisted respondents on Part B questions on behalf of EPA during the detailed 

questionnaire response period, while maintaining communications throughout the active survey 

data collection period with EPA and other EPA support contractors. When ERG received a 

question from EPA, ERG responded either by email from the ERG CBM Helpline email address 

or contacted the respondent directly, depending on whether the questionnaire respondent’s 

contact was made by email or by phone. After contacting the respondent, ERG entered the 

question and response on a contact log sheet. For those questions that required additional input 

from EPA, ERG noted that fact on the contact log sheet. This task has been completed.

2.3.3 Receipt and Internal Tracking of Questionnaires

ERG set up a survey tracking form and procedures to quickly identify issues with Part B 

portions of questionnaires and to ensure the expeditious routing of questionnaires through the 
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 6 of 9

review, coding, and data entry phases (see Appendix A, which presents a portion of the ERG 

survey tracking form). This task has been completed.

2.3.4 Developing a Survey Database and Data Dictionary

Under Work Assignment 6-05, ERG developed a database for entering questionnaire 

responses for Part B of the questionnaire, as well as a data dictionary describing the data 

elements. In Work Assignment 8-05, ERG made minor revisions to these items. In addition, 

ERG developed procedures to use for performing QA/QC of the data and preparing and printing 

data tabulations. This task has been completed.

2.3.5 Reviewing, Transcribing, Coding, and Entering Respondent Data into Survey 

Database

For the detailed questionnaire, ERG reviewed each survey for missing information that is 

needed prior to data entry. ERG also identified those respondents requesting assistance with 

transcribing their financial information from tax forms and other financial reports into the 

questionnaire format. If requested by EPA, ERG will telephone respondents whose surveys were 

missing information (see Section 2.3.6). Following the review process, ERG coded responses on 

the hard-copy questionnaires to ensure that the responses were in a form that can be easily 

entered into the detailed survey database. Using forms developed in Access for this purpose, 

ERG then entered the data. These procedures have been developed so that complete, legible, 

permanent, and defensible records documenting all work performed are found within the hard 

copy questionnaires and in the database itself. An example of the detailed work required of ERG 

staff during the review, transcription, coding, callbacks and data entry process can be seen in 

Section 3.4.2 and Appendix A (Instructions for Survey Tracking, Review & Coding). This task 

has been completed.

CBM operators are able to identify responses containing confidential business 

information (CBI) on the questionnaires. ERG is handling all CBI data in accordance with 

applicable EPA procedures (see Section 2.5).

2.3.6 Following Up on Incomplete Surveys

As ERG received the detailed questionnaires, they were reviewed for completeness (i.e., 

to ensure that responses to all questions were provided). A very few questionnaires were 

identified as having insufficient data and would require callbacks prior to data entry. Others, 

more nearly complete, were allowed to go through data entry before callbacks are made.

Procedures for contacting respondents are identified in Section 3.4 of this QAPP. ERG has

developed documentation for the public record that summarizes how the questionnaires were 

reviewed, coded, and data entered. ERG will call respondents with incomplete surveys only 

under technical direction from EPA. If ERG calls respondents, we will document the call and

any revisions made to the database as a result of the call (see Appendix A). To date, ERG has not 

called any respondents for missing data.
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 7 of 9

2.3.7 Tabulating Data 

Once all the questionnaires were entered into the survey database, in accordance with the 

procedures in Section 3.4, ERG conducted and continues to conduct accuracy checks and will 

follow up further on any problems identified in reviewing the hard-copy questionnaires and data, 

as requested by EPA. ERG also is identifying whether the database contains any outliers and will 

investigate them accordingly. The criteria for making any changes to outliers in the database are 

discussed in Section 6.0.

2.4 Quality Objectives and Criteria (Element A.7)

The primary objective of this QAPP is to ensure that the data collected using the CBM 

detailed questionnaires will be of the quality necessary to support EPA in determining whether it 

would be appropriate to revise the Oil and Gas Point Source Category ELGs to control pollutant 

discharges from CBM produced water. The tasks described earlier, in Section 2.3, involved the 

collection and use of data gathered using a detailed survey of selected CBM projects. 

Questionnaire data collected directly from industry personnel are considered primary data. ERG 

defined the acceptance criteria (i.e., data specifications) it used to evaluate whether the data 

collected under the economic portions of the screener and detailed questionnaires would be 

suitable for EPA’s study needs. The acceptance criteria were related to the completeness, 

accuracy, and reasonableness of questionnaire responses. 

2.4.1 Completeness

Completeness was defined as the number of valid responses received compared to the 

number of questionnaires distributed. As described in Part B of the ICR Supporting Statement 

(Appendix B), EPA considered non-responses to specific questions and non-responses to the 

overall detailed questionnaires in developing the sampling design for the CBM industry. 

Therefore, acceptable completeness was determined in terms of EPA’s statistical sampling plan. 

ERG will work with EPA’s statisticians to determine the completeness required for specific 

questions or within the sample selection strata. The specific methods ERG is using to improve 

completeness are described in Section 3.4 of this QAPP. 

2.4.2 Accuracy

Questionnaire accuracy pertains to:

• Whether the respondents provided correct answers to the questions asked; 

• Whether the responses provided were interpreted correctly by the questionnaire 

sponsor; and 

• Whether the responses entered into the questionnaire database were free from 

error. 

ERG was not tasked with developing and implementing tools, such as conducting on-site 

inspections and interviews, to measure elements 1 and 2. The methods ERG is using to measure 
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 8 of 9

element 3, and to increase the accuracy of responses, are described in Section 3.4 of this QAPP. 

ERG’s goal for data entry accuracy was 100 percent error-free. 

2.4.3 Reasonableness

To be suitable for use in EPA’s overall CBM Detailed Study, the data collected by the 

detailed questionnaires must reasonably reflect economic and financial realities and industry 

practices. For example, a small project with 4 wells is highly unlikely to be producing billions of 

cubic feet of gas per year; likewise a CBM project would not have received a wellhead price of 

gas in 2008 of $100/million Btus (MMBtu), which historically have ranged generally between $2 

and $10/MMBtu in the U.S.). ERG’s goal is to ensure that all responses are reasonable. The 

methods ERG uses to evaluate and improve the reasonableness of questionnaire responses are 

described in Section 6.0 of this QAPP.

2.5 Special Training Requirements/Certification (Element A.8)

As noted previously, the owners and operators of CBM industries could indicate that 

certain responses on their questionnaires contained CBI. ERG’s Document Control Officer 

(DCO) for CBI is working in conjunction with EPA’s DCO to regulate access to all CBI received 

by ERG under this work assignment. Only authorized ERG employees are allowed access to the 

CBI. Only the ERG DCO or an authorized employee can release CBI material.

Access to CBI by ERG employees is based on an employee’s need-to-know status. 

Before access is authorized, an employee’s justification for access is documented and submitted 

to the ERG DCO. This notice indicates the name of the employee, why access to CBI is 

necessary, and the Work Assignment number under which the employee uses CBI. The ERG 

DCO informs the employee of his or her responsibilities for the security of CBI. The employee is 

required to familiarize himself or herself with the appropriate procedures for handling CBI and is 

required to pass a test of CBI handling procedures specific to ERG’s Lexington Office. 

ERG submitted a CBI security plan for the Lexington Office (where the CBM CBI 

financial materials are held under this work assignment) to EPA in 2006; this plan, which was 

approved by EPA at that time, remains active for all effluent guidelines-related support. 

2.6 Documentation and Records (Element A.9)

ERG developed and instituted document control mechanisms for reviewing, revising, and 

distributing QAPPs. Each QAPP has a signed approval form, title page, table of contents, and 

EPA-approved document control format, which appears in the upper right corner of each page. 

(See header at the top of each page of this QAPP for a specific example.) During the course of 

the project, ERG circulates any revisions to the QAPP to technical and QA/QC project staff at 

ERG and EPA. 

To finalize and implement the detailed questionnaire and the Economic Analysis task,

ERG has collected information and data and will prepare summaries of those data (additional 

data collection that is currently ongoing is discussed in Section 6.0). These collected data and 
Section No. 2

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 9 of 9

summaries include questionnaire responses, helpline questions and responses, questionnaire 

tracking documentation, and summary memoranda related to the survey effort. In addition, 

survey responses have been entered into a database for subsequent analysis by EPA. Memoranda 

and reports that will be generated under the Economic Analysis Task are discussed in more detail 

in Section 6.0. All technical data collected, generated, and analyzed on the project are handled 

and stored in a centralized file in ERG’s Lexington, Massachusetts, office, located in a room

dedicated to CBI materials. All documents collected or produced in the course of the project are 

stored in the centralized file or (if not CBI) in designated files held by the ERG Economics Task 

Manager or her designates. The ERG Economics Task Manager designated an economics project 

staff member to manage and maintain all CBI project files in the Lexington Office CBI room. 

She also ensures that appropriate files are submitted to ERG Chantilly (where the ERG WAM 

manages all rulemaking record compilations for the CBM industry under Work Assignment 9-

37) at periodic intervals throughout the assignment. 

ERG describes how it managed data obtained from the CBM detailed questionnaires in 

Section 3.4 of this QAPP under Element B.10, Data Management. Computer files containing the 

survey data are maintained by ERG employees on CDs or flash drives that are used (but not 

stored) on dedicated, non-networked CBI computers as required by ERG’s Lexington Office CBI 

plan. An ERG staff member designated as database manager prepares CD or flash drive backups 

of all survey data daily throughout the data entry process and whenever updates to the database 

are made following the data entry process. 

As described in Section 2.5 above, materials designated as CBI are maintained under 

restricted access and used only in accordance with ERG’s policies and procedures for CBI by 

staff who have received CBI clearance, with oversight by the Lexington and Chantilly DCOs.

All quality system documents and records will be maintained for the life of the contract 

or until EPA requests that ERG destroy the records or return them to EPA. Quality system 

records were filed according to guidance provided by the ERG Program Manager and Project 

QA Coordinator. 

Disposition of quality system documents and records will be determined in accordance 

with contract requirements. 
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 7

3. DATA GENERATION AND ACQUISITION ELEMENTS

The detailed questionnaire involved collecting and reviewing data obtained from the 

owners and/or operators of CBM extraction projects. Elements B.1 to B.3 and Element B.10 

below are relevant to this task. ERG’s approach to each of these elements is described in the 

subsections that follow.

• Element B.1, Sampling Process Design

• Element B.2, Sampling Methods

• Element B.3, Sample Handling and Custody

• Element B.4, Analytical Methods

• Element B.5, Quality Control

• Element B.6, Instrument/Equipment Testing, Inspection, and Maintenance

• Element B.7, Instrument/Equipment Calibration and Frequency

• Element B.8, Inspection/Acceptance of Supplies and Consumables

• Element B.9, Non-direct Measurements

• Element B.10, Data Management

3.1 Sampling Process Design (Element B.1)

As discussed in Sections 1.0 and 2.3 of this QAPP, EPA conducted a detailed survey of a 

sampling of CBM projects identified with the screener questionnaire. Using a sample frame of 

773 projects (screener surveys), EPA, with support from Battelle, selected 203 CBM projects to 

receive detailed questionnaires. EPA selected the projects for sampling by basin, establishment 

size, and discharge method (i.e., direct or indirect discharge, and zero discharge). Additional 

details on the sampling process design have been provided in a memorandum prepared by 

Battelle under a separate contract (EPA Contract No. EP-05-030, Work Assignment 4-11: 

Coalbed Methane (CBM) Study: Final Detailed Questionnaire Sample Design) (6). An 

additional 20 projects were selected on the basis of a judgment sampling methodology, as 

outlined in the memorandum titled Selected Projects for the San Juan Judgment Sample (7). This 

task has been completed.

3.2 Sampling Methods (Element B.2)

EPA distributed the detailed questionnaires under the authority of Section 308 of the 

Clean Water Act (Federal Water Pollution Control Act, 33 U.S.C. § 1318). All CBM project 

operators that received a detailed questionnaire were required by law to respond to it. This task 

has been completed.

3.3 Completeness 

As it relates to the sampling method, completeness was defined as the number of valid 

responses received compared to the number of questionnaires distributed. Because responses are 

legally required, EPA assumed a relatively high response rate for the survey effort. For the 

detailed questionnaire, EPA adjusted the estimated sample size assuming a non-response rate of 

10 percent (for more details, see EPA Contract No. EP-05-030, Work Assignment 4-11: Coalbed 
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 2 of 7

Methane (CBM) Study: Final Detailed Questionnaire Sample Design) (6). This assumption of a 

10 percent non-response rate was based on EPA’s experience with a typical effluent guidelines 

questionnaire. EPA will adjust the detailed questionnaire sample weights for any non-response 

for the survey.

When developing the initial drafts of the detailed questionnaire EPA and ERG worked 

closely with applicable industry trade associations. The CBM industry was, therefore, generally 

well informed of the goals and objectives of EPA’s study effort and of EPA’s schedule for 

questionnaire distribution and receipt. This effort was intended, in part, to improve completeness.

Under a previous work assignment, ERG and EPA had developed contact names and 

mailing lists for CBM operators. An accurate mailing list reduced the number of undeliverable 

questionnaires and improved completeness. 

3.3.1 Accuracy and Reasonableness

For the questionnaires to result in accurate information concerning the CBM industry, 

respondents had to clearly comprehend the questions asked and provide responses that accurately 

reflect their operations. ERG and EPA then must interpret those responses correctly. Under an 

earlier work assignment, ERG and EPA worked with industry representatives to determine the 

best terminology to use when developing questions. Industry representatives also made 

suggestions regarding the types of questions that would be clear to most respondents. For 

example, EPA responded to comments that small businesses might not be able to understand the 

form in which firm-level data was being requested and inserted a question allowing small 

businesses to provide tax forms or accounting reports in lieu of responding to many of the firmlevel financial questions. ERG transcribed such financial information into the questionnaire for 

respondents that requested this approach. ERG and EPA also incorporated wording changes, 

provided definitions for key terms, and incorporated other suggestions from industry so that draft 

questions would elicit the information needed to answer questions related to EPA’s overall CBM 

Economic Analysis Task. 

The entities listed below provided comments on drafts of Part B of the detailed 

questionnaire, including the terminology used, the definitions of key terms, and the availability 

of the data requested: 

• Internal ERG Technical Reviewers;

• EPA core workgroup members;

• EPA extended workgroup members ;

• CBM industry and trade association personnel;

• General public, including tribes, environmental groups, and other stakeholders; 

and

• The Office of Management and Budget (OMB).

These steps have been completed. The reasonableness of responses is currently being assessed. 

This task is discussed in Section 6.0.
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 3 of 7

3.4 Data Handling and Custody (Element B.3)

Under Contract No. 68-C-02-095, ERG Lexington prepared files and tracking 

spreadsheets using the survey identification number, and all communications regarding the 

survey used the survey identification number to ensure the proper identification of all survey 

respondents and connect them to the records of those communications. ERG Lexington and ERG 

Chantilly use these identifiers in communications under Contract No. 68-C-02-095, Task 09-37.

ERG established an electronic system to track each detailed questionnaire internally at 

the ERG Lexington office, from receipt of the copy through all stages of review, coding, data 

entry and callbacks. The system included the unique survey identifier and the dates the 

questionnaires were received in Lexington, reviewed, coded, or assigned a callback requirement. 

Appendix A shows the form used to track the surveys. 

To document reviewer mark-ups, ERG Lexington used the photocopies of detailed 

questionnaires sent for this purpose (with originals held at ERG Chantilly). These photocopies 

are stored in ERG’s CBI room in files arranged by the unique survey identifier number and 

respondent name. Any coding or revisions to data entered into these copies were made in nonblack pen, color coordinated to an individual reviewer, who documented the pen color by 

entering his/her signature on a reviewer cover sheet or by signing his/her initials in the review 

copy. In this way, ERG documented which reviewers made which changes, and changes can be 

distinguished from the original responses easily. The review and coding tasks are complete. ERG 

can provide EPA with pdfs on CD-ROM or DVDs of the marked-up photocopies for the public 

record (minus any data identified as CBI), if requested. ERG, under a separate task on this work 

assignment, is maintaining the public record for the entire CBM survey effort. 

ERG managed the data entry process for Part B of the detailed questionnaire. For this 

process, ERG reviewed, coded, and entered the Part B detailed questionnaire data into the 

previously constructed database. This task is complete. (ERG’s activities related to review and 

data entry of Parts A and C of the detailed questionnaire are being performed under other tasks 

under this work assignment and have a separate QAPP as noted in Section 1.0.) ERG is adhering 

to EPA procedures for handling CBI when handling and transmitting questionnaires.

3.5 Data Management (Element B.10)

ERG has implemented and continues to implement the following QA activities to 

improve the completeness, accuracy, and reasonableness of responses to the detailed 

questionnaires received. 

3.5.1 Questionnaire Non-Response Follow Up

ERG Lexington will conduct follow-up (e.g., telephone respondents), if requested by 

EPA, when respondents submitted surveys but failed to complete Part B of the questionnaires. 

ERG Chantilly is responsible for follow-up, if requested by EPA, when respondents did not

return any part of the survey. ERG has provided information to EPA detailing which respondents 
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 4 of 7

are refusing to provide firm-level or project-level financial information for further follow-up. 

EPA will decide which of several other means for collecting questionnaire data should be 

implemented, including, as necessary, additional follow-up phone calls and letters, criminal 

fines, civil penalties, and other sanctions, as provided by law. ERG also operated a helpline 

(through telephone or email systems) where CBM industry owners and operators could submit 

questions regarding the surveys and ERG coordinated with EPA to respond to these inquiries in a 

consistent manner. The helpline task is complete. These activities helped to improve the 

completeness of responses on individual questionnaires (see Section 3.3).

3.5.2 Questionnaire Response Review and Follow Up

ERG Lexington reviewed Part B of each received detailed questionnaire. This initial 

review evaluated completeness and reasonableness. Certain questions if not completed, or if a 

large number of key questions were incomplete, were identified as triggering a potential need for 

an initial callback prior to data entry (for example, if no Part B was submitted) or for subsequent 

callbacks after data entry (if certain key information was missing). As necessary, and if requested 

by EPA, ERG will call respondents when questionnaires are found to be substantially 

incomplete. After the initial review, ERG conducted a more detailed review. This review 

identified: 

• Missing or incomplete responses not necessitating callbacks prior to data entry 

during initial Part B review and tracking phase;

• Inconsistent responses (e.g., questions completed when a response directed that 

they be skipped, responses that contradicted other questionnaire responses); and

• Unreasonable responses (out of the permissible range).

ERG is also conducting a comprehensive evaluation following data entry. Database 

queries are being used to identify inconsistent and/or unreasonable responses, as well as issues 

such as line items not summing to totals reported. These activities are ongoing (see Section 6.0

for more details relating to this evaluation). If any of these issues are identified, if so directed by 

EPA, ERG will call the respondent to obtain a complete, consistent response to each question. 

ERG will document those communications and change the original questionnaire response based 

on the follow-up communication. The communications will be logged, and the communication 

record will be stored in the survey file in ERG’s CBI-secured room. After review and follow-up 

are completed, ERG will report the number of valid responses provided for each question, so that 

EPA statisticians will be able to determine the accuracy of the estimates made using those 

responses.

To further improve the completeness and accuracy of questionnaire responses in Part B, 

ERG and EPA provided assistance to questionnaire respondents by telephone and an email 

messaging system. For the detailed survey effort, ERG also offered support to respondents in 

completing the detailed questionnaire. For small businesses who requested the help, ERG 

transcribed firm financial data from tax forms and accounting reports submitted by respondents

into the questionnaires. 
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 5 of 7

Before the detailed questionnaires were mailed, ERG also developed FAQs that 

contained anticipated questions from questionnaire respondents and appropriate responses from 

ERG and EPA staff. ERG routinely discussed with EPA the nature of questions being received 

and the answers being given to ensure that bias was not being introduced into the survey process. 

ERG updated the FAQs as necessary and distributed the updated FAQs to appropriate project 

team members.

ERG has completed the documentation of the following with respect to the detailed 

questionnaires: 

• Receipt dates; 

• Specifics regarding follow-ups to helpline and email inquiries; and

• Findings from initial and detailed review of completed questionnaires.

ERG will document or continues to document the following with respect to the detailed 

questionnaires:

• Notes from any follow-up inquiries with respondents and related edits made to 

questionnaire responses; and 

• Any changes made to questionnaires or the database. 

ERG used or will use a form (Figure 3-1) to document telephone conversations with 

respondents. Those forms documenting any callbacks made to respondents that might be 

requested by EPA during the data analysis or tabulation stages, will be stored in the applicable 

questionnaire file folder in the CBI room for the study. Forms documenting Helpline calls are 

stored electronically on ERG’s network (no CBI helpline calls have been received at ERG). ERG

will provide copies of these records to EPA as requested. A more detailed discussion of the 

survey tracking, handling, review, coding, and callback procedures is provided in Appendix A. 

3.5.3 Questionnaire Response Coding and Data Entry

ERG developed a Microsoft Access database in which responses to the detailed 

questionnaires were entered. Responses were entered using Access Forms developed specifically 

for the database structure to minimize data entry errors. 

Before entering the data for Part B of the detailed questionnaire, ERG staff coded the 

completed detailed questionnaires to ensure that all responses were in the format required for 

data entry. (Questionnaire coding improves the accuracy of data entry.) A CBI-cleared ERG 

employee with prior experience related to questionnaire data entry keyed information from the 

hard-copy surveys into the database using the Access Forms. Each data entry staff member was 

required to briefly review their entries as they completed each form for error checking purposes. 

To further ensure that the database is accurate, a second CBI-cleared ERG staff member 

independently reviewed all of the questionnaires for data entry accuracy. If data entry errors 

were identified, ERG staff logged the changes necessary in a form and made the change to the 
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 6 of 7

data. All changes were rechecked by another staff member. To further minimize the potential for 

keying or other errors, all data entry staff were selected based on their experience with using 

Microsoft Access. When errors in database entry are discovered during any reviews, ERG 

corrected the erroneous data fields. ERG staff not involved in the data entry corrections 

performed a 100 percent check to ensure that resolved errors were correctly re-entered into the 

Microsoft Access database. Similar 100 percent checks are also being or will be performed for 

any changes made to the database on the basis of reasonableness changes for follow up (see 

Section 6.0) or callbacks to the survey respondents.

Project Name: CBM Economic Analysis Task

Project No: 

Facility ID: 

Facility Name: 

Contact Name: Contact Phone No: Contact Email Address: 

Date: Staff Name: 

Time: AM PM

This document 

contains CBI: 

_____ Yes

_____ No

Type of Contact:

 Email Telephone 

General Subject(s) 

Discussed: 

CBI =

Yes (Y)

Question

No. Topics Discussed

Figure 3-1. CBM Detailed Questionnaire Contact Record

The completed database containing detailed questionnaire Part B data is stored 

electronically on flash drives in ERG’s secure CBI files. 

After all responses are added to the database, ERG is executing a series of queries to 

determine whether any responses were missing, did not add to a reported total, were outliers, or 

were otherwise problematic for a given question for any questionnaire respondent. ERG will

follow up on any such identified discrepancies, if requested by EPA, by contacting respondents.
Section No. 3

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 7 of 7

Any changes made to the database to correct erroneous entries will be handled as discussed in 

Section 6.0.
Section No. 4

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 3

4. ASSESSMENT AND OVERSIGHT ELEMENTS

This section describes the methods ERG used to assess the quality of work conducted for 

the questionnaire data collection tasks.

4.1 Assessments and Response Actions (Element C.1)

The ERG WAM, ERG Economics Task Manager, and ERG project QA Coordinator

appoint Technical Reviewers as needed during the conduct of the work assignment. The products 

that received a technical review included one or more of the following: 

• Revisions to the detailed questionnaire and

• Draft/final survey database and database dictionary.

The products that will receive a technical review include:

• Formal memoranda, such as those outlining methodologies or methods for 

replacing missing or erroneous data in the Detailed Questionnaire; and

• Any draft economic analysis report chapters or economic analysis report delivered 

to EPA. 

ERG will document all reviews of project deliverables in an electronic tracking 

spreadsheet. All deliverables will subsequently be reviewed by ERG’s Economics Task Lead and 

the ERG WAM. 

ERG performed a technical review of the documentation of follow-up work on 

incomplete questionnaires. Technical reviews are critical evaluations of work performed. ERG

adhered to procedures for conducting and documenting the review, coding, and callback process 

as outlined in detail in Appendix A during the questionnaire review. Each Questionnaire Part B 

received two separate reviews by at least two different staff members. The first reviewer 

reviewed and coded the questionnaire and the second reviewer ensured that the reviewing and 

coding was complete, accurate, legible, and consistent with the reviews of other questionnaires. 

ERG will continue to follow these procedures if EPA requires additional follow-up with 

questionnaire respondents.

Following these two reviews, a sampling of questionnaires with identified data 

completion issues and all supporting documentation was reviewed by a Technical Reviewer (a 

staff member other than the reviewers performing the review and coding tasks) to gauge 

adherence to the procedures outlined in Appendix A and to ensure that all documentation was 

complete. In general, technical reviews are conducted by professionals who are proficient in the 

work areas of interest, but who are not directly responsible for performance of the work. 

Technical reviewers document the results of their reviews. For this work assignment, the ERG 

Economics Task Manager is responsible for ensuring that technical reviews are conducted by the 

appropriate personnel and are documented. These reviews have generally been completed, 
Section No. 4

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 2 of 3

although if any callbacks are made, additional review of callback documentation and data 

changes will also be performed.

ERG’s Project QA Coordinator assesses the implementation of QA/QC procedures on 

this project as follows:

• Reviewed this QAPP for completeness and applicability and signed the signature 

page of the QAPP following successful review(s). 

• Ensure project staff understand and follow the QC procedures in this QAPP prior 

to continuing work on the project.

• Audit project files to verify that project staff are (1) using the QC procedures 

identified in this QAPP and (2) documenting the deliverable review process. If 

not, the ERG QA Coordinator will direct the ERG WAM and/or ERG Economics 

Task Manager to implement corrective action and document that the action was 

taken. Corrective action could include additional staff instruction or the 

development and use of additional measures to ensure the quality of data 

management and review.

4.2 Reports to Management (Element C.2)

The ERG WAM informs the EPA WAM of the progress of all work activities through the 

monthly progress reports required under Work Assignment 9-37 of Contract No. 68-C-02-095.

ERG’s Technical Reviewers and project QA Coordinator communicate any quality 

deficiencies identified as a result of reviews or audits to the ERG WAM in writing. If quality 

objectives and criteria are not met, the ERG WAM is responsible for ensuring that the 

appropriate corrective actions are taken and reported to ERG’s QA team. An example of an 

appropriate corrective action would be additional data entry review if data entry errors are 

identified. 

As required by Work Assignment 9-37, ERG will electronically submit to the EPA WAM 

and PO a monthly progress report that documents the costs incurred and work performed during 

the previous accounting period and work planned for the current accounting period. Additionally, 

the ERG WAM will provide the EPA WAM with monthly reports on the status of quality 

assurance (QA) activities. These reports will be incorporated into our monthly technical progress 

reports which will be supplemented to include the following: 

• Quality assurance activities performed during the reporting period as part of the 

implementation of the work assignment;

• Identification of any problems encountered;

• Identification of any deviations from the QAPP; and

• Identification of problem resolution and/or corrective actions taken during the 

reporting period, if any.
Section No. 4

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 3 of 3

ERG will describe QA activities conducted for major deliverables, such as summary memoranda, 

in the Technical Progress Reports (TPR). If data do not meet all the quality specifications but are 

needed for analysis, ERG will obtain approval to use the data from EPA and will reference the 

approval in the QA documentation in the TPR.
Section No. 5

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 1

5. RECONCILIATION WITH USER REQUIREMENTS

Completed detailed questionnaires are the primary data sources that EPA will use to 

answer the key questions posed in Section 2.2.1 of this QAPP. The questionnaire responses will 

provide detailed descriptions of CBM extraction operations, industry finances, project 

economics, and produced water management practices. EPA will use the questionnaire data, in 

combination with secondary data, to support its overall CBM Economic Analysis Task. 

ERG is thoroughly documenting for the public record all primary and existing source 

data used in any reports or memoranda developed. Subsequent analyses of questionnaire 

responses will clearly describe for the record any limitations of the quality of the questionnaire 

data or assumptions required for its use as described in Section 6.0. Project deliverables will also 

clearly identify the sources of secondary data collected on this project. The deliverables will 

clearly describe the limitations of the data; for example, when information from a source may be 

dated, ERG will note in the presentation of conclusions that the source may have data quality 

limitations. However, unless otherwise noted, all data used will meet the currency, accuracy, 

reliability, and scope requirements specified in Section 6.2. 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 16

6. ADDENDUM COVERING THE DATA ANALYSIS AND USE OF EXISTING DATA IN THE 

COALBED METHANE ECONOMIC ANALYSIS

6.1 Introduction

This section covers six new tasks that will be conducted as part of the CBM rulemaking:

1. Task 8.a: SISNOSE assessment and recommendation

2. Task 8.b: Develop information for SBREFA evaluations

3. Task 8.c: Develop impact methodologies

4. Task 8.d: Prepare calculations to analyze preliminary engineering inputs

5. Task 8.e: Evaluate profile data

6. Task 8.f: Recommend approaches for missing data

This addendum does not modify the quality assurance procedures discussed in the first five 

sections of this document but incorporates additional discussion of the new tasks. Please refer 

back to Section 1.0 of this QAPP for a description of the CBM project.

In response to guidelines issued by the Office of Management and Budget (9) under 

Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 

(Public Law 106-554; H.R. 5658), EPA developed Information Quality Guidelines (8) for 

ensuring and maximizing the quality of information disseminated by EPA. EPA’s Information 

Quality Guidelines defines quality as:

• Objectivity - information is accurate, reliable, and unbiased and its presentation is 

clear, complete, and unbiased;

• Utility - information is useful for the intended users; and 

• Integrity - information is not compromised through corruption or falsification 

either by accident or by unauthorized access or revision (10)

In executing projects involving the collection and use of existing data, ERG will support 

EPA in meeting these guidelines by:

• Understanding how EPA intends to use the collected data and collecting data 

appropriate for that use;

• Working with EPA to develop specifications for the required accuracy, reliability, 

and objectivity of the data, evaluating collected data against those specifications, 

and including the results of these evaluations in intermediate and final project 

deliverables; and

• Presenting the sources of data, assumptions made, and calculations used in all 

project deliverables. These presentations will be sufficiently detailed and 

transparent to ensure the reproducibility of the work by qualified third parties. 

This section describes how ERG will ensure the quality of our collection and use of 

economic data by using the four-step plan-do-check-report cycle repeatedly until the project is 

completed, as summarized below. ERG will collect quality information as we conduct the project 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 2 of 16

and report that information with the project deliverable(s). Table 6-1 presents the steps of this 

four-step process.

Table 6-1. The Four-Step Plan-Do-Check-Report Cycle

PLAN

• Identify key questions/objectives

• Identify the type and quality of information required to answer the questions/meet the 

objectives

DO • Collect the required data

CHECK • Evaluate collected data against the quality specifications

REPORT • Report results of the data quality evaluation

PLAN

• How to use the collected data

• Analysis and/or summary of collected data

• Required quality of the analysis results

DO • Conduct the planned analysis

CHECK • Evaluate analysis results against the quality specifications

REPORT • Report results of the quality evaluation of the analysis

The remainder of this QAPP section is organized into the sections below. Please refer to 

Section 2.0 for the overall project objectives and management structure.

• Section 6.2: Project Description and Objectives

• Section 6.3: Data Needs, Sources, and Quality Criteria

• Section 6.4: Data Acquisition, Review, Quality Verification, and Reporting

• Section 6.5: Use of Existing Data

• Section 6.6: Project Reports (Deliverables)

6.2 Project Description and Objectives

ERG is supporting EPA in assessing the impacts of the costs of candidate treatment 

technology options for CBM produced water. Please refer back to Section 1.0 and Section 2.2 of 

this QAPP for a general description of the CBM project. The purpose of this section is to 

describe how ERG will use the detailed questionnaire data, identified and reviewed according to 

Sections 2.0 through 5.0 of this QAPP, and additional existing data which is described in 

Sections 6.3, 6.4, and 6.5 of this section. ERG will use these data to develop estimates of impacts 

on CBM projects, firms and other affected entities using EPA-supplied estimates of costs for 

CBM produced water treatment technology options. In addition, this addendum describes how 

ERG will identify and extract data subsets from the regulatory database for economic and 

statistical analysis at the request of the EPA WAM.

ERG will determine which projects are considered not financially viable (for example, 

have negative earnings) in the baseline analysis using data in Part B of the detailed questionnaire 

submitted by respondents. Such data include 2008 volumes of gas produced by project, average 

project wellhead price received for gas in 2008, operating costs, and other expenditures. ERG 

will use engineering estimates of technology option costs that will further reduce financial health 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 3 of 16

to identify how many projects incrementally become non-viable financially when technology 

option costs are considered. To the extent feasible, ERG will investigate sources of existing data 

that might allow more recent financial conditions to be assessed. Additionally, the detailed 

questionnaire data will be analyzed to identify if any CBM firms are considered likely to fail in 

the baseline or incrementally after costs for technology options are used to adjust financial line 

items as reported. To the extent possible, these line items and project finances will be adjusted 

for current financial conditions using existing data sources. Revenue data from the detailed 

questionnaire will be analyzed for small affected firms to determine if technology option costs 

could pose a significant impact on a significant number of small entities (SISNOSE). Section 

6.5.1 discusses the quality criteria for identifying methodologies for making these 

determinations. EPA will evaluate the results of these impact analyses to determine which (if 

any) of the technology options represent Best Available Technology Economically Achievable 

(BAT).

6.3 Data Needs, Sources, and Quality Criteria

ERG is undertaking an analysis of the impacts of possible alternative produced water 

management methods on CBM operators and their projects using the following primary data, 

which will be the source of much of the financial data that will be used to characterize firms and 

projects:

• Screener Survey database analyses of numbers and business size of operators, 

wells, projects, gas production, and method of produced water management. This 

database was prepared under the QAPP for the Coalbed Methane Detailed Study 

Questionnaire, dated January 27, 2010 and approved by EPA March 17, 2010, 

written by PG Environmental, Inc. [11]).

• The Detailed Questionnaire, Part B, database analyses of firm finances, project 

finances, and other data that will be used to calculate impacts on firms and 

projects. This database was prepared under Revision 0 of this QAPP, approved on 

November 4, 2010, as represented in Sections 1.0 through 5.0 of this current 

document.

• Any additional data voluntarily provided by CBM operators that reflect changes 

from 2008, typical gas and water decline rates by basin, or other information, as 

requested by EPA. These data will be handled similarly to existing sources data 

using the procedures outlined in 6.5 and 6.6 of this QAPP, including tracking, 

review, and data entry into applicable input databases, proofing, and additional 

reviews.

ERG will also be adding new existing data sources under the current Work Assignment 

9-37, including:

• Data collected from Securities and Exchange Commission (SEC), state oil and gas 

commission websites, and CBM operator websites to augment and update profile 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 4 of 16

and other data —SEC filings and CBM operator websites are expected to provide 

limited updated information on firm finances and size, while the state oil and gas 

commission websites will provide information on current levels of gas and water 

production, production decline rates, and any changes in ownership or operator 

status among the surveyed CBM operators.

• Additional CBM production cost data prepared by the Department of Energy 

(DOE) to augment cost data in basins where missing data might be an issue.

• DOE data on current state average wellhead prices for gas to provide information 

for calculating revised project revenues using more recent production data by 

operator obtained from state oil and gas commission websites.

• Compilation of SEC data from publicly owned domestic oil and gas firms 

prepared by Oil and Gas Journal to assist in updating the profile beyond the 2008 

date of the survey data and to compare publicly held CBM operators to the wider 

group of oil and gas firms in the industry.

• Production cost indices from DOE, which will add to the discussion of natural gas 

production cost trends. These indices are available from DOE as a graph, which 

will be downloaded intact with explanatory text for the graph summarized.

• DOE’s Annual Report, to provide projected wellhead prices in the U.S. and other 

background information on the future of natural gas generally and CBM 

specifically over the next 30 years.

• Other information gathered to provide data to assist in developing replacements 

for missing data, as key missing data points are identified.

• Any other information identified as useful during the course of the data gathering 

and analysis or as requested by EPA.

This section describes how ERG will acquire, review, and document the existing data. 

Table 6-2 lists the criteria that ERG will use to evaluate if existing data source and/or 

questionnaire data are acceptable for use in analyses supporting the CBM rulemaking. ERG will 

provide information on the quality of data sources that may be used in analyses, and the EPA 

WAM will make the final determination of whether the source is suitable for use. If ERG 

identifies limitations in the quality of the data from data sources that may affect decisions 

regarding the overall direction of further data collection efforts or the quality of deliverables, 

ERG will communicate these issues to the EPA WAM. After a data source is determined to be 

acceptable for use, ERG may use the data to complete the rulemaking analyses described in 

Section 6.5. Table 6-3 discusses the criteria that ERG will use to evaluate data for use in 

replacing missing or erroneous data. Finally, Table 6-4 presents the criteria that ERG will use for 

accepting economic impact estimates.
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 5 of 16

Table 6-2. Industry Profile and Updated Production and Financial Data Acceptance 

Criteria

Acceptance Criterion: 

Literature 

Review/Profile/Financial 

Data Information Description/Definition Specification

Currency (up to date) Does the information reflect 

current conditions?

• Sources were published or presented in 2001 or after. 

• Because CBM is a relatively new industry, ERG may 

use information from sources prior to 2001 if it 

represents practices currently in place. ERG will 

document all data collected (see Section 6.3). The 

EPA WAM will provide further direction on how to 

incorporate or exclude the data in analyses.

Accuracy/Reliability Is the information true? ERG assumes that the underlying data and information 

contained in the following sources are accurate:

• Government publications (state and federal reports);

• Government-maintained websites and databases;

• Peer-reviewed journal articles;

• Presentations at professional technical conferences; 

and

• Publications from recognized industry experts or 

organizations. The EPA WAM will determine 

recognized experts.

ERG assumes that data from these sources are of 

sufficient quality to evaluate for use in CBM rulemaking 

activities. ERG will, however, note if these sources 

themselves make any disclaimers.

• The data are peer reviewed and well respected in the 

industry, such as the Oil and Gas Journal.

Company websites for employment information and 

parent firm-subsidiary relationships are considered 

acceptable sources for privately held firms.

For other sources, ERG will determine, case-by-case if 

the source contains information that appears to be 

accurate. 

Regardless of the source, ERG will provide EPA with 

the following information and EPA will provide further 

direction on how to incorporate or exclude the data in the 

analysis:

• Source citation information;

• Analysis methodology and assumptions (when 

applicable); and

• Data limitations or analysis uncertainty.

Scope Is the information reported 

from CBM operations?

• Information pertains to CBM operations and not 

strictly to conventional oil and gas or shale gas.

• Source represents a geographic area in which CBM 

operations are active or emerging.
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 6 of 16

Acceptance Criterion: 

Literature 

Review/Profile/Financial 

Data Information Description/Definition Specification

Reasonableness Is the information consistent 

with EPA’s understanding 

of CBM operations? 

• ERG will compare information collected from 

existing data sources to primary sources (e.g., 

questionnaire data) to confirm the information is 

reasonable based on documented CBM practices. This 

comparison will be presented in the QA sections of 

major deliverables.

Table 6-3. Criteria for Acceptance of Replacement Data for Missing or Erroneous Data

Acceptance Criterion: 

Missing or Erroneous 

Data Description/Definition Specification

Currency (up to date) Does the information reflect 

current conditions?

• The source for replacement data will primarily be the 

2008 Detailed Questionnaire. Additional data will be 

considered, but 2008 or newer data will be selected 

preferentially. 

Accuracy/Reliability Is the information true? • Erroneous data will be identified in the Economics 

Database by identifying outliers compared to other 

responses and using the knowledge and judgment of 

the CBM industry or by running calculation checks to 

identify summing errors in financial data.

• First choice for replacement data will be drawn from 

other responses to the Detailed Questionnaire or from 

corrected summations of the financial data.

• Other choices for replacement data might include data 

associated with standard industry practices (such as 

typical severance tax rates by state). 

Scope Is the information reported 

from CBM operations?

• Information pertains to the basin or state in which the 

operator is producing.

• If firm-related, information pertains to similar size 

firms operating in the basin or state in which the firm 

is producing or is the same information found publicly 

(e.g., SEC filings for that firm).

Reasonableness Is the information consistent 

with EPA’s understanding 

of CBM operations? 

• Erroneous data will be identified based on 

understanding of CBM operations as well as 

comparisons to other responses in the Economics 

Database.

• Replacement data for both erroneous and missing data 

will reflect what is known about typical operational 

conditions among CBM operations by basin
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 7 of 16

Table 6-4 Impact Estimates Quality Specification 

Acceptance Criterion: Economic Impact 

Estimates Specification

Meet requirements for EPA decision making • Include baseline and incremental counts of firm 

failures and project closures, along with other impacts 

that EPA might request.

• Provide appropriate annualizations of compliance 

costs by technology option, aggregated by basin and 

nationally

Standard year(s) used for impact analyses 

matches year or years used for engineering 

cost analyses 

• All revenue and cost data from the detailed survey are 

to be updated using appropriate measures of inflation 

(Producer Price Indices or Consumer Price Indices, 

depending on the line item) to 2008 and to 2010.

Complete • All projects and firms currently discharging 

wastewater and are subject to new proposed 

requirements will be analyzed.

• Non-respondent firms and projects will be analyzed 

either based on statistical weighting or matching to 

similar firms/projects.

• New projects will be developed based on information 

provided for newer projects in the affected basins.

Accurate • Calculations reflect the criteria outlined in the 

approved economic impact methodologies and 

replacement data reflect the data as proposed in 

approved memos suggesting data replacements.

Reasonableness • Firm failures, significant firm impact, or project 

closures identified in analyses are consistent with 

review of the changes calculated in the key financial 

data (e.g., compliance costs exceed available cash 

flow so firm failure is likely regardless of changes in 

other financial data). 

ERG will prepare a methodology memorandum that discusses potential impact analysis 

approaches (Task 8.c) and a series of additional memoranda that will or might be needed to 

address Task 8.a (SISNOSE assessment and recommendations), Task 8.b (develop information 

for SBREFA evaluations), Task 8.e (evaluate profile data), and Task 8.f (recommend approaches 

for missing data). The EPA WAM will approve these memoranda and select the methodologies 

to be used for the firm and project economic analyses prior to ERG’s development of economic 

impact results. These memoranda are discussed in more detail in Section 6.6. 

6.4 Data Acquisition and Review

ERG staff will use the following procedures when collecting published existing data:
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 8 of 16

• ERG economists will collect full content and bibliographic information on all 

newly collected publications, including “date accessed” for information collected 

from the Internet. ERG researchers will save web pages as PDF files, so that the 

data source can be added to the electronic project file.

• ERG economists will provide bibliographic information for use in the master 

project file, housed in Chantilly, VA, in a form that will allow the procedures 

described in the project file management plan to be followed (Appendix A of the 

Quality Assurance Project Plan for the Coalbed Methane Existing Data 

Collection [2]).

• ERG economists will submit the publications and meta data to the project file 

index keeper as described in Appendix A of the Quality Assurance Project Plan 

for the Coalbed Methane Existing Data Collection (2). 

ERG staff will use the following procedures to document data collected via telephone and 

email contact.

• ERG researchers will document all telephone contacts, questions asked, and 

responses using a Telephone Contact Report, such as the one depicted in Figure 3-

1 in Section 3.5.3 of this QAPP. 

• ERG researchers will save email contacts as PDF files, so that they can be added 

to the electronic project file.

• ERG researchers will ensure that telephone contact reports and email records are 

added to the CBM Project File.

As discussed in Section 3.5.3 of this QAPP, ERG will store economics questionnaire data 

collected for the CBM project in the economics portion of the CBM regulatory database in 

Lexington, MA (and in the overall CBM regulatory database in Chantilly, VA, as well). 

Additionally, ERG will store, separately from the economics questionnaire database, existing 

data collected for the CBM project, and additional input databases developed from a combination 

of the questionnaire data and existing data sources, as applicable. 

ERG will evaluate existing data sources listed in the tracking database. ERG will also 

discuss the procedures for pulling a data subset for input databases, which will include any 

portion of data from the regulatory database that the EPA WAM requests, for use in project 

analyses.

6.4.1 Existing Data Tracking Database (Tracking Database) and Development of Detailed 

Data Tables

ERG will store citation and review information for existing data sources in a tracking 

database which will be used in conjunction with the project file index. The tracking database 

summarizes information beyond the bibliographic information required for the record index. For 

each source, the tracking database will identify topics covered, citation information, and any 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 9 of 16

additional sources mentioned in the source. Most of the fields in the tracking database are simple 

checkbox or yes/no fields that indicate the type of information included in the source. The 

tracking database does not store actual data, but rather identifies whether or not the source 

discusses particular topics. ERG will use the fields in the tracking database to identify potential 

sources that may have useful data to contribute to the rulemaking analysis. The tracking database 

will indicate the location of any data copied into spreadsheets or other data formats. Existing data 

sources may not include operator and project level data, but they will contain information on 

specific basins, such as production declines or current wellhead prices, that can be used to 

supplement the primary data collected in the detailed questionnaire. ERG will evaluate the 

existing sources using the criteria specified in Section 6.3 and provide information on the quality 

of sources to the EPA WAM. The EPA WAM will determine whether the source is suitable for 

use in the rulemaking analysis.

ERG will identify data sources that may be used during the rulemaking analysis. These 

sources may have information relating to current financial conditions, operators who have ceased 

producing, operators who have sold projects to others, production declines in basins, basin-wide 

production information, and other information useful for characterizing the industry. ERG will 

enter the data into detailed data tables in a format appropriate for use in analyses, such as 

Microsoft Excel or Access. To review the accuracy of the compilation of the detailed data tables, 

the ERG Economics Task Lead will assign a project team member to compare at least 20 percent 

of the entries in the detailed data tables to the original data sources. If the project team member 

finds errors, an ERG project team member will document the errors and conduct additional 

checks of the data (the extent will depend on the severity of the original errors and the outcomes 

of the additional checks). 

6.4.2 Development of Data Subsets

ERG will extract data subsets from the regulatory database at the request of the EPA 

WAM. The EPA WAM will request specific data for economic analyses and ERG will create 

database queries to pull the specific information requested. 

ERG will verify that the data housed in the regulatory database has been reviewed 

according to the procedures outlined in Section 3.0 and Section 6.3 of this QAPP. ERG will 

provide a description and purpose of the data subset as well as documentation of the data subset. 

ERG will document the data subset by adding a description to all field names within the data 

subset and by providing a database dictionary to accompany the data subset. ERG will also 

document the query development and QC procedures used to create the data subset. All of this 

documentation will be included in a separate table within the regulatory database, which will 

include the following fields:

• Query name (for data subset pull);

• Name of ERG staff who created query;

• Date query was created; 

• Name of ERG staff who completed QC check;

• Date QC check was completed; and
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 10 of 16

• Additional notes.

Because the primary data are CBI, data subsets are CBI materials and ERG will manage 

their creation and distribution as specified in ERG’s 2005 Security Plan for Handling CWA 

Confidential Business Information, Lexington, MA Office (12).

6.5 Use of Existing Data and Quality Verification

ERG will use collected data to estimate impacts on CBM operators resulting from 

incurring costs for changes to their management, treatment, and disposal systems using the 

candidate technology options selected by EPA. ERG will also estimate impacts specific to small 

firms in the industry. ERG will document the use of existing data for project analyses in 

memoranda that include:

• Purpose of analysis;

• Description of analysis methodology;

• Data sources used;

• Criteria used to select the data sources;

• Exceptions to the criteria and rationale for exceptions; and

• Documentation of assumptions and equations. 

ERG will check the accuracy of the analyses using the procedures outlined in Section 6.4 

of this addendum, which discusses the verification and validation methods.

The following sections describe use of data and quality verification according to the fourstage plan-do-check-report cycle.

6.5.1 Develop Methodologies and Evaluate Profile Information (Tasks 8.a, 8.c, 8.d, and 8.f)

ERG will implement the four-stage plan-do-check-report cycle to develop the economic 

impact estimates. The first stage in the four-part cycle to develop the estimates is to “plan.” 

ERG will develop methodologies for computing impacts on small businesses, all firms, projects 

(existing and new), employment, communities, and the production of natural gas. 

ERG will work with the EPA Economics Task Leader in developing methodologies for 

using the data contained in the economic databases that reflect sound analytical approaches for 

assessing impacts. The methodology will identify each of the fields that will be used as inputs to 

economic analyses. For the SISNOSE evaluation (Task 8.a), the criterion will be to use the EPAaccepted screening approach for evaluating small business impacts, which looks at compliance 

costs as a percentage of revenues. The two major criteria for the selection of additional 

recommended methodologies will be: 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 11 of 16

• What methodologies are commonly used by the oil and gas industry themselves or 

the financial investment industries for evaluating the financial viability of oil and 

gas projects and oil and gas firms; and 

• What methodologies have been previously used in other ELG analyses related to 

the oil and gas industry (e.g., those used for the synthetic drilling fluids ELG 

[13])?

Methodologies previously used to analyze the economic impact of ELGs established for other 

subsectors of the oil and gas industry have withstood the notice and comment rulemaking 

process. However, the two criteria above can be met with the same methodology, although there 

may be some minor modifications needed to address the few differences between conventional 

oil and gas projects and CBM projects.

After the EPA Economics Task Leader approves the methodologies, ERG will develop a 

memorandum (as part of Task 8.c) that describes, in detail, the assumptions, data inputs, 

equations, algorithms, outputs, outliers and testing procedures that will be used to estimate 

economic impacts. ERG will compute these impacts following the procedures approved by 

EPA.

Another part of the “plan” step is to evaluate the data to be used to create the economic 

profile (Task 8.b). ERG will work with the EPA Task leader in identifying the key pieces of 

information needed to create a comprehensive profile of the industry, including a means of 

updating the profile to account for industry changes in the intervening years between survey data 

and a draft profile. (Note that many more analyses can be prepared for a CBI profile than can be 

performed for a non-CBI profile.)

The criteria for selecting key pieces of profile data include the types of data that are 

generally included in all economic profiles used for ELGs to characterize the size, structure, and 

financial health of the industry and the types of data that are pertinent to characterizing the 

distinct features of the affected industry. These distinctive features were addressed in the 

Detailed Study Report and are expected to be expanded using information now available from 

the Detailed Questionnaire. Additionally, certain data that are used to develop broader 

assumptions in the impact analyses might be highlighted in the profile. Also, because the 

industry has changed significantly since 2008, which is the date of the data in the Detailed 

Questionnaire and Screener, ERG will be investigating sources of information to update the

profile. These existing data will be collected using the criteria shown in Table 6-2. 

ERG will provide a detailed outline of a profile chapter to the EPA Economics Task 

Leader. After approval of the contents of the profile chapter, ERG will develop the profile using 

the data compiled from the detailed questionnaire and existing data. Examples of the types of 

data likely to be used in the profile include:

• Total and average amounts of CBM gas and water produced by basin, by 

discharging and non-discharging projects (in 2008 and updated); 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 12 of 16

• Total numbers of non-discharging and discharging firms in each basin (in 2008 

and updated);

• Total numbers of non-discharging and discharging projects in each basin (in 2008 

and updated);

• Counts of small and large firms by non-discharging and discharging status (in 

2008 and updated);

• Average operating costs for CBM wells by basin and by discharging and nondischarging projects;

• Changes in the firms operating CBM projects since the survey was undertaken;

• Average revenues of discharging firms vs. non-discharging firms, by basin and by 

size of firm (small vs. large); 

• Counts of firms by corporate structure types.

• Average reported interest rates and hurdle rates.

• Average taxable earnings by discharging vs. non-discharging firms and by small 

vs. large firms, by basin.

• Any changes to financial information since 2008 among publicly held firms 

(dischargers vs. non-dischargers, small vs. large, to the extent publicly available 

data are available).

• Average construction costs of new projects (those begun in 2004 or later).

• Other information as requested by the EPA Economics Task Lead or the EPA 

WAM.

All profile information will be evaluated on the basis of criteria outlined in Table 6-2. 

An additional part of the “plan” stage is the evaluation of missing and erroneous data and 

making recommendations for data replacement. ERG will prepare memos documenting these 

evaluations and recommendations as a part of the planning process.

6.5.2 Calculate Impacts 

The second stage in the four-part cycle to develop the economic impact estimates is to 

“do.” ERG is supporting EPA’s decision on the economic achievability of technology options. 

ERG will evaluate the ability of firms and projects to absorb the incremental costs of the 

technology options and determine, using decision criteria specified in the methodologies 

approved by the EPA Economics Task Leader whether the firms and projects can afford the 

incremental costs. In conducting this task, ERG will use data from the economics database 

created from the Detailed Questionnaire as discussed in previous sections of this QAPP, as well 

as existing data sources such as state oil and gas commission websites, SEC filings, corporate 

websites, and other sources identified through literature and internet searches and evaluated as 

discussed in Section 6.2.
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 13 of 16

The calculation of economic impact will follow the equations and algorithms outlined in 

the methodology memo (Task 8.c) and discussed above in Section 6.5.1. In calculating 

economic impacts, ERG will:

• Following EPA’s approval of recommendations for replacing data (Task 8.f), create the 

replacement data and input the data into the data subsets created as discussed in Section 

6.4.2 ;

• Develop the projection data needed to provide a rough estimate of project financial 

conditions out ten years, as will be described in the methodology memo;

• Incorporate the projection data and other input data into the spreadsheet calculation tools, 

subject to the criteria in Section 6.3 and the steps discussed in Section 6.5.3;

• Calculate economic impacts for options selected by EPA; and

• Create tables for EPA review, also subject to the steps discussed in Section 6.5.3. 

At least three calculation tools will be created: a SBREFA tool, a firm impact tool, and a 

CBM project impact tool. The calculation tools (created in Excel or Access) will use the input 

data created in the subset databases to determine numbers of firms considered likely to fail 

incrementally, numbers of small businesses likely to incur costs that exceed 1 percent and 3 

percent of revenues, and numbers of projects likely to shut in during the first year. The 

calculation tool for project impacts can also be used to determine reductions in overall project 

life and production, and identify impacts on federal and state tax collections and impacts on 

mineral rights owners. The calculation tool for firm failures and that for project closures can 

also be used to identify the potential impact on employment. The economic impact analysis 

document will record the following elements:

• Electronic tool description;

• Flow charts (if necessary);

• Data sources;

• Documentation of assumptions;

• Discussion of any boundary or limiting conditions;

• Documentation of any default parameter values and their sources; and

• Documentation of all equations (and sources, if applicable) used to generate 

output.

6.5.3 Check Results of Impact Estimates

The third stage in the four-part cycle to develop the economic impact estimates is to 

“check.” ERG will perform a QC check of the Excel or Access tools created for estimating the 

firm and project economic impacts. The QC check will verify the appropriateness of the 

approach and verify that original data are correctly input. The QC check also will verify that the 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 14 of 16

tools implement the documented approaches (including those for data replacement). A CBM 

project team member not involved in the calculations will review any constants used for 

appropriateness, will check that units are converted properly, and will document and check 

equations used against the documentation and references. To the extent possible, the team 

member will independently verify 20 percent of the calculations using hand calculations or 

separate spreadsheets for three representative CBM projects. If any checks fail, then the 

spreadsheet will be revised and retested. The ERG economics task leader and an ERG senior 

reviewer will review the results of the calculations to evaluate whether the results make sense 

(e.g., are on the proper scale, follow expected trends, or are comparable to data reported by 

industry) and provide EPA with information appropriate for use in rulemaking decisions.

ERG will document the electronic tool development in database tables or spreadsheets 

that include:

• A list of any data changes including a description, the purpose, and the process used to 

change the data;

• The name of the ERG staff member who performed the QC and the date completed; and

• A list of any errors found during QC and the subsequent resolution(s).

When changes are made to the electronic tool, ERG will:

• Assign a unique name and number to the revised version of the tool (in either Excel or 

Access); 

• Maintain a historical file of the changes; and 

• In a QA spreadsheet, log the changes made to the different versions and explain what was 

revised, the date the revision was made, and by whom. 

ERG will determine whether the economic impact estimates meet the specifications 

presented in Table 6-4 by comparing the estimates to primary data from the questionnaires and 

existing data. ERG will verify and validate the estimated costs and loads by comparing them to 

actual costs and loads for similar projects. ERG will present any uncertainties about the estimates 

to the EPA WAM, who will make the final determination about whether they are appropriate for 

use in the rulemaking analysis.

Table 6-5 provides a summary of the procedures ERG will use to determine that data 

used and results generated on this project meet quality specifications.

Table 6-5. Summary of Procedures and Results Quality Verification

Action Task QA Actions

PLAN Develop methodology Before delivery to EPA, an ERG senior economist will provide 

conceptual review of methodology and assumptions. The EPA WAM 

will approve the final methodology.

Develop calculation approach

Evaluate replacement data for 

missing or erroneous data
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 15 of 16

Action Task QA Actions

DO Insert approved replacement 

data

A CBM team member who did not do the calculation will check 

equations against the documentation and references. To the extent 

possible, the team member will independently verify 20 percent of the 

calculations using hand calculations or separate spreadsheets for three 

representative CBM projects for the project impact tool. Additionally 

all calculations will be checked for 20 percent of the discharging 

operators for the SBREFA and the firm impact tools. If the checks fail 

(i.e., any errors are found), then the spreadsheet will be revised and 

retested. The ERG Economics Task Leader and an ERG senior 

reviewer will review estimates to evaluate whether they are appropriate 

for the CBM industry.

Calculate baseline project and 

firm financial conditions 

(closures and failures)

Annualize Technology Option 

costs 

Aggregate project costs by 

option to the operator level

Insert appropriate annualized, 

O&M and/or capital costs, 

depending on the tool and 

methodology 

Calculate numbers of 

incremental significant impacts 

on small businesses, project 

closures, firm failures, and other 

impacts as specified by EPA.

CHECK Methodology and assumptions The ERG Economics task leader and a CBM team member who did 

not do the initial calculations will review the creation and use of 

electronic tools. An ERG senior reviewer will review calculation 

results to evaluate whether they are appropriate for the CBM industry.

Spreadsheet and/or database 

uses the methodology and 

assumptions

Option cost data are properly 

loaded

Data replacements are properly 

specified and loaded

Results make sense (on both 

project and overall level)

Results are similar to existing 

projects and other sources

REPORT Develop project reports (see 

Section 6.6)

The ERG Economics Task Leader and an ERG senior reviewer will 

review all project reports and evaluate whether they are appropriate for 

the CBM industry. Additionally, the ERG WAM will also review all 

project reports. ERG will include the results of the economic impact 

analyses, the QA procedures, and the results of the quality review in 

the project reports.

6.6 Project Reports (Deliverables)

The fourth stage in the four-part cycle that ERG will use to develop the costs and loads 

calculations is to “report.” Table 6-6 lists the key products ERG will produce and how data 

quality will be reported in each. 

ERG will include an evaluation of data quality in all project deliverables. ERG will also 

identify the sources of data, assumptions made, and calculations used in their development in all 
Section No. 6

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 16 of 16

project deliverables. These identifications will be sufficiently detailed and transparent to ensure 

the reproducibility of the work by qualified third parties. All project deliverables will meet 

EPA’s standards of transparency, objectivity, integrity, and utility as specified in EPA’s 

Information Quality Guidelines (8).

Table 6-6. Key Products

Key products Content

Regulatory database • Complete the portion of the regulatory database that comprises the economics 

questionnaire database and the tracking database. 

Data subsets • Relevant subsets of the regulatory database for use by other parties in analyses 

outside of this work assignment.

Firm, project, and other 

methodologies

• Explains estimating baseline and compliance financial conditions at firms and 

projects and identifying firm failures, project closures, and other impact 

measures.

Firm and project impact 

analysis detailed assumptions

• Assumptions, equations, inputs, algorithms, outputs, testing procedures, etc. 

used to estimate impacts on firms, projects, and other entities.

Impact estimates • Estimates of impacts of technology options on firms, projects and other 

entities.
Section No. 7

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 1 of 2

7. REFERENCES

1. ERG (July 13, 2011). Quality Assurance Project Plan for the Review and Data Entry of 

the Engineering Portions of the Coalbed Methane Questionnaire. Contract No. 68-C-02-

095.

2. ERG (August 25, 2011). Quality Assurance Project Plan for the Coalbed Methane

Existing Data Collection. Contract No. 68-C-02-095.

3. ERG (November 30, 2011). Data Analysis Addendum to the Quality Assurance Project 

Plan for the Coalbed Methane Existing Data Collection. Contract No. 68-C-02-095.

4. Battelle (November 2, 2007). Quality Assurance Project Plan for WA 2-11: Statistical 

Support for Coalbed Methane Detailed Study; Contract No. EP-C-05-030. 

5. U.S. Environmental Protection Agency (March 2001). EPA Requirements for Quality 

Assurance Project Plans, EPA QA/R-5. EPA, Office of Environmental Information, 

Washington, DC, EPA/240/B-01/003.

6. Battelle (October 19, 2009). EPA Contract No. EP-05-030, Work Assignment 4-11: 

Coalbed Methane (CBM) Study: Final Detailed Questionnaire Sample Design.

7. ERG (2009). Selected Projects for the San Juan Judgment Sample. Memorandum from 

Anne Jones, ERG, to James Covington, III, EPA. October 21.

8. U. S. EPA (2002). Guidelines for Ensuring and Maximizing the Quality, Objectivity, 

Utility, and Integrity, of Information Disseminated by the Environmental Protection 

Agency. EPA/260R-02-008. October.

9. OMB (2002). U.S. Office of Management and Budget. Guidelines for Ensuring and 

Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated 

by Federal Agencies (67 FR 8452).

10. EPA (2003) U.S. Environmental Protection Agency. A Summary of General Assessment 

Factors for Evaluating the Quality of Scientific and Technical Information, EPA 100/B03/001. June, 2003.

11. PG Environmental, Inc. (January 27, 2010). QAPP for the Coalbed Methane Detailed 

Study Questionnaire. dated 2010 and approved by EPA March 17, 2010, written by PG 

Environmental, Inc.

12. ERG (2005). Security Plan for Handling CWA Confidential Business Information, 

Lexington, MA Office. October.
Section No. 7

Revision No. 1

Date: March 15, 2012

QAPP—CBM Detailed Questionnaire Page 2 of 2

13. EPA (2000).Economic Analysis of Final Effluent Limitations Guidelines and Standards 

for Synthetic-Based Drilling Fluids and other Non-Aqueous Drilling Fluids in the Oil and 

Gas Extraction Point Source Category
QAPP—CBM Detailed Questionnaire

Appendix A

INSTRUCTIONS FOR SURVEY REVIEW & CODING
QAPP—CBM Detailed Questionnaire

A-1

INSTRUCTIONS FOR SURVEY TRACKING, REVIEW & CODING

Prepared for:

James Covington

U.S. Environmental Protection Agency

Engineering and Analysis Division

Office of Water

1200 Pennsylvania Avenue, NW

Washington, D.C. 20460

 Prepared by:

Eastern Research Group, Inc.

110 Hartwell Avenue

Lexington, MA 02421

Updated March 10, 2010
QAPP—CBM Detailed Questionnaire

A-2

INSTRUCTIONS FOR SURVEY TRACKING, REVIEW & CODING

Introduction

ERG-Lexington is responsible for tracking, reviewing, coding, and data-entering Part B 

of the Coalbed Methane Survey (Financial Information). Appendix A provides the Operator ID, 

name of responsible person and contact information for each respondent from which we expect a 

survey. 

We will be receiving copies of the complete survey by overnight courier (we will only be 

responsible for review and coding of Part B, however). All surveys, until otherwise identified as 

not containing CBI information, will be handled as CBI materials and must be logged into the 

CBI filing system either by Anne Jones, Sara Giberson, or their designate. Once the package is 

opened, the appropriate tracking sheet column (see Figure 1 at the end of these instructions, 

which shows a portion of the tracking spreadsheet) must be completed for each Survey ID that is 

not considered out of scope (in addition to the CBI logging paperwork). The full tracking sheet is 

stored electronically in I://econ/coalbed 2007.aj/survey/detailed questionnaire/survey tracking & 

review/tracking spreadsheet.xls. Once the CBI log-in and the initial tracking sheet log-in is 

completed, the survey questionnaire will be placed in a prepared wallet packet, labeled with the 

Survey ID and Respondent Name. A review packet specific to the survey respondent with a 

review cover page and completeness checklist for each project we are expecting to be covered in 

the survey will also be inserted into the wallet packet (see Appendix B). The survey will then be 

ready for an Initial Review.

The tracking data sheet will be kept on the desktop of the middle computer in the CBI 

room. On each Friday morning, the latest version of the tracking data sheet will be reviewed, 

copied and uploaded to the network by Sara Giberson, database manager. The database manager 

will then summarize the status of the surveys in an email to the ERG WAM (Anne Jones). For 

example, a status report might indicate that 35 percent of surveys have been received; of these, 

all have been logged in, 75 percent have been reviewed and coded, 60 percent have been 

reviewed, coded, and placed in the data entry file, and 10 percent have data entry completed.

Review

There are two parts to the review process. The first part ensures that Section B is present, 

that financial data has been provided (either in the questionnaire itself or as an attachment) and 

that no projects are missing. It also distinguishes surveys with additional work needed to 

transcribe data provided by the respondents (i.e., financial reports and/or attachments with 

information relevant to financial projections. The reviewer fills out the Initial Review Sheet (see 

Appendix C, Form 1) at this stage. The reviewer will also check off any issues, as noted, on 

Form 3 (Appendix C). This form is simply a summary of issues and notes across all surveys with 

initial issues (or special data attachments). Only those surveys with the issues or data indicated in 

Form 3 need to be inserted on Form 3. It will serve as a means of summarizing how the Initial 

Review is proceeding and will ensure that data that will not be inserted into the questionnaire 

will be identified as needing to proceed to the contractor in charge of gas and water projections.
QAPP—CBM Detailed Questionnaire

A-3

The second stage of the review process investigates whether the survey is reasonably 

complete and readable. A survey may have a few missing pieces and still be considered 

reasonably complete and ready for data entry; however, after data entry, callbacks might be 

necessary. Some surveys might have gross completeness problems beyond those identified in the 

Initial Review or readability issues (see Form 2, Appendix C). These surveys will require 

callbacks prior to any attempts at data entry. A checklist on Form 2 will assist in an assessment 

of whether the survey is complete enough to go through data entry prior to callbacks. If any of 

the listed questions on Form 2 are missing answers, it will probably be best to call the recipient 

back prior to data entry; if there are any doubts, the data manager will make a decision. Surveys 

with key pieces of missing information might be held until review and coding of relatively 

complete surveys are finished. All others will go to coding. Following coding, a second review 

will be performed to ensure the completeness review and coding are correct and readable. At that 

point, surveys will be ready for data entry. The following sections present the details of these 

steps. Section 2.1 discusses the initial review and financial data transcription, while Section 2.2 

covers the completeness reviews.

All reviewers and their roles will be assigned in advance on a sheet posted in the CBI 

room. These assignments will be reviewed periodically by the ERG Task Manager, and, if 

necessary, reassignments will be made based on availability and the inclusion of additional staff. 

Pen colors for each staff member will also be posted on the assignment sheet.

Initial Review and Financial Data Transcription

The following presents the steps to be taken following the wallet packet setup steps 

outlined above. The labeled wallet should contain all survey materials, including possible 

attachments and cover letter, and the reviewer packet (Appendix B). 

1. Select a survey wallet and checks to see if the respondent notes that they are out of scope 

(they did not produce any CBM in 2008). To do this, check Table A-2 in Part A. 

a. Proceed to Step 2 if survey is in scope (Table A-2 has wells listed for at least one 

project).

b. If Part A is missing or Table A-2 has no entries, check the cover letter or the 

comments to Part A to see if respondent has noted no CBM was produced.1

c. If not in scope, the Initial Review Sheet (see Form 1) is checked “Yes” for “Out 

of Scope.” 

d. Note on Form 3 that the survey is out of scope and place it in the out of scope file 

located in Drawer 2 of File Cabinet E.

2. Check to see that Part B is present.

a. If present, proceed to Step 3.

b. If not, check “Yes” for “Missing All of Part B” on Form 1.

c. Note on Form 3 that the survey is missing Part B.

 1 We may have several of those, because several respondents were not sent the Screener Survey and could not opt 

out at that time. 
QAPP—CBM Detailed Questionnaire

A-4

d. File the survey in the “Initial Problems” file, in the fourth drawer of File Cabinet 

D. Surveys in this file drawer require further completion of the initial review once 

Part B is received.

3. Check Question B2-10. 

a. If “Yes” is checked, mark “No” for missing financials (the respondent was not 

required to submit any firm financial information) and proceed to Step 4.

b. If “No” is checked, check Questions B2-17 through B2-23.

1. If at least one year’s information appears in the answers to most of these 

questions, mark “No” for missing financials and proceed to Step 4.

2. If no information is contained in these questions but financial reports are 

attached, mark “No” for “Financial Data for B2 Missing” and mark 

“Yes” for “B2 Financial Data Needs Transcribing” and proceed to Step 

4.2

3. If none of these for any years are completed AND no financial reports 

are attached, check “Yes” for “Financial Data for B2 Missing” on Form 

1. 

4. Make a note on Form 3 for that survey ID that financial reports are 

missing. Proceed to Step 4. 

5.

4. Check Table A-2 against the projects listed in the review packet. There should be one 

Table A-2 for each project as well as one page labeled Part B3 in the completeness 

review packet in Appendix B for each project we were expecting the respondents to 

include in the survey). 

a. If project names and count of projects match between Table A-2 and the review 

packet, proceed to Step 5.

b. If project names differ between the review sheet and Table A-2, or there are more 

or fewer projects than expected, check the comments to Part A, the cover letter, or 

comments to Part B for any explanation. 

1. If the explanation is sufficient, mark “No” for Projects missing:

1. Adjust the review forms to match (discard omitted project review 

sheets, add new project review sheets, and/or change project names 

on the review sheets). 

2. Contact the ERG Task Manager (Anne Jones) if there is any 

concern about the sufficiency of an explanation. 

3. Contact Anne Jones to obtain a project number for any new 

projects. 

4. Proceed to Step 5.

2. If there is insufficient explanation, mark “Yes” for “Projects Missing” 

on Form 1 and note this in Form 3 for that survey ID. Proceed to Step 5.

5. Check the last box of Question B3-75 or boxes under Question B3-80 and B3-81 noting 

gas or water curves or projections have been attached are X’d 

 2 We will transcribe data even if they were supposed to have done this themselves (answered No to Question B2-15, 

as long as they sent financial reports).
QAPP—CBM Detailed Questionnaire

A-5

a. If attachments referenced are attached (or if any attachments relevant to 

projections whether or not the boxes indicated are checked), note on For 3 that 

projection data is attached. Proceed to Step 6.

b. If projection information should have been attached (above referenced boxes were 

checked, but no attachments were found), check “Gas & Water Projections are 

Missing” on Form 1 and make a note of this on Form 3. Proceed to Step 6.

6. Review Form 1 for all items marked “Yes”.

a. Surveys with boxes marked “Yes” for Issues 3 (missing financial reports) and/or 5 

(missing projects) are placed in the “Initial Review Problem” file area. Proceed to 

Step 7 when directed by the ERG WAM.

b. If Issue 4 is marked “Y” and Issues 3 (missing financial reports) and 5 (missing 

projects) are both marked “No” (Issue 6 can be “Yes”), the survey is placed in the 

Financial Data Transcription file area (Drawer 1 of Cabinet C) with a green dot 

affixed to the wallet. Skip to Step 8.

c. If all issues but Issue 6 are marked “No,” the survey is placed in the Active File 

(Drawers 3 and 4 in Cabinet E), with a blue dot in place on the wallet noting that 

initial review is complete. That is, missing projection data, while a significant 

problem, does not interfere with continuing the review, coding and data entry 

phases. You are done with the Initial Review.

7. Callbacks will be required to resolve Issues 2, 3, 5, and 6. 

a. Contact the respondent, asking for the information that is lacking.

b. Inform the respondent that additional callbacks might be required at a later stage 

of review.

c. Complete a telephone log of your conversation.

d. When information arrives, data manager will place information in the appropriate 

wallet and revise Form 1.

e. When all issues on Form 1 have been resolved (with the exception of Issue 6—

missing projections), ensure all Form 1 information is updated and information 

and proceed with the remainder of the review process:

1. If Part B was missing and has been received, continue with 

Step 3.

2. If financial reports or projects were missing and have been received, 

continue with Step 6.

3. If missing projections are received, file them with the survey, update 

Form 1, and note on Form 3 that projections are available.

8. Transcription:

a. Two economists will work on the transcription of financial data. The first will 

transcribe the data, the second will review and proofread the transcriptions. 

b. Financial Data Reviewer 1:

1. Pulls out a survey from the Transcription File

2. Writes in the Survey ID and initials and dates Form 4 (see Appendix C), 

which is posted in the CBI room, in the “Transcription Begun” Column. 
QAPP—CBM Detailed Questionnaire

A-6

3. Completes the appropriate section of the Review Cover Sheet for the 

Survey (see Appendix B) using any pen color but black.

4. Inputs information from the financial reports directly into B2 using the 

same pen color used to sign the Review Cover Sheet.

5. Completes the transcription.

6. Returns the survey and review packet to the wallet

7. Initials and dates Form 4 in the “Transcription Completed Column.”

c. Financial Reviewer 2:

1. Checks Form 4 for completed transcriptions

2. Pulls a transcribed survey from the Transcription File.

3. Initials and dates Form 4 in the column “Review of Transcription 

Begun” for the survey ID pulled.

4. Completes the appropriate section in the Review Cover Sheet in a 

different pen color than used for the initial transcription.

5. Proofs all transcriptions using the same pen color to mark any changes.

6. Confers with transcriber, data manager, or ERG Task Manager if 

necessary to resolve discrepancies and to finalize questions for any 

possible callbacks.

7. Places any callbacks needed, informing respondents that another 

callback may be needed prior to data entry when further survey reviews 

are completed.

8. Fills out telephone logs to document changes made as a result of 

conferring with respondent.

9. Inputs all final changes into Section B2.

10. Places the survey and review packet, and any phone logs back into the 

wallet

11. Affixes a blue dot on the wallet and places the survey in the Active File 

(Drawers 3 & 4 in Cabinet E).

12. Initials and dates Form 4 (“Transcription Complete”). You are done with 

Initial Review.

9. Data summary reports (every Friday morning):

a. The database manager reports which survey IDs have been placed in the problem 

file and why and which are missing projection data, using Form 3. 

b. The data manager reviews Form 4 for “Transcription Complete” and transfers this 

information to the survey tracking form.

c. The data manager transfers the copy of the survey tracking spreadsheet from the 

CBI room to the network for review by the ERG WAM.

Detailed Review

Two persons will undertake each detailed review of surveys that have a blue dot affixed 

to the wallet. The steps of the detailed review are as follows:

Detailed Reviewer 1

1. Pulls out a survey with a blue dot from the Active File.
QAPP—CBM Detailed Questionnaire

A-7

2. Initials and dates Form 5 (see Appendix C) in the appropriate column, which will be 

posted in the CBI room.

3. Opens the reviewer packet (see Appendix B) and fills in the Reviewer Cover Sheet using 

a pen color different from those used by Financial Reviewer 1 and 2. 

4. Turns to the Completeness Review Checklist for B1 and B2 (each survey ID has one such 

sheet) and the Completeness Review Checklist for B3 (there is one sheet for each 

project).

5. Using the same pen color with which the Cover Sheet was signed, goes through all the 

questions in Part B of the questionnaire, indicating what, if any missing items are found 

by checking off and making notations in the Completeness Review Checklist.

6. If, at any point in the Review 1 process, the reviewer needs to place the survey back in 

the active file, circles the “Review 1 Begun” box for that survey ID on Form 5 and inserts 

a note into the wallet explaining where in the process the review stopped. 

7. Transcribes all issues to be discussed in a callback on Form 6 (See Appendix C), the 

Reviewer Flag/Comment Sheet.

8. Reviews Form 2 (Initial Callback Trigger Form) and compares it to missing items in the 

checklist to see if the survey will need an initial callback. Checks “Yes” if any of the key 

information is missing in the Form 2 checklist.

9. Alerts the Data Manager to any issues checked on Form 2.

10. Discusses the extent of the key missing data (one missing item might be acceptable; 

several should be considered a sign that the respondent requires an initial callback). Data 

manager notes a decision to make an initial callback in the tracking spreadsheet.

11. If an initial callback is required:

a. Returns the survey, review packet, all relevant Forms (including Form 6) to the 

wallet

b. Places a red star on the wallet.

c. Places the wallet in the “Detailed Review Problems” file area (Fourth Drawer, 

Cabinet C). Callbacks are discussed in more detail in Section 4.

12. If an initial callback is not required:

a. Initials and dates “Review 1 Complete” on Form 5. 

b. Affixes a silver star to the wallet and returns the survey and all associated forms 

to the wallet and places the survey in the Active File AND/OR

c. Initials and dates “Coding Begun” on Form 5.

d. Begins the coding process (see Section 3, below).3

e. At the completion of coding, notifies the data manager that the first review and 

coding is complete so that the information can be entered into the survey tracking 

spreadsheet.

If coding must be interrupted, 

circles the “Coding Begun” column for that ID on Form 5 and returns the wallet 

to the active file.

f. Initials and dates “Coding Completed” on Form 5.

g. Affixes a red dot on the wallet to indicate that the first review and coding are 

complete. 

h. Returns the wallet to the Active File (Drawers 3 & 4, Cabinet E).

 3 Note: if necessary, a person other than Reviewer 1 can code, but no checklist or coding should be reviewed by the 

same person who completed the checklist or coding. The coder signs into the review cover sheet as well.
QAPP—CBM Detailed Questionnaire

A-8

Detailed Reviewer 2

1. Pulls out a survey with a red dot from the Active File.

2. Initials and dates Form 5 in the appropriate column, which will be posted in the CBI 

room.

3. Opens the reviewer packet (see Appendix B) and fills in the Reviewer Cover Sheet 

using a pen color different from those used by Financial Reviewer 1 and 2 and 

Detailed Reviewer 1 (and/or Coder). 

4. Turns to the Completeness Review Checklist for B1 and B2 (each survey ID has one 

such sheet) and the Completeness Review Checklist for B3 (there is one sheet for 

each project).

5. Using the same pen color with which the Cover Sheet was signed, goes through all 

the questions in Part B of the questionnaire, indicating what, if any changes must be 

made to the Completeness Review Checklist or to the coding instructions. 

6. Refers any legibility issues or discrepancies needing resolution to the Data Manager.

7. Checks the transcription of all issues to be discussed in a callback on Form 6, the 

Reviewer Flag/Comment Sheet.

8. If, at any point in the Review 2 process, the survey is returned to the active file prior 

to completion, circles “Review 2 Begun” on Form 5 and inserts a note into the wallet 

explaining where in the process the review stopped. 

9. At the completion of the second review, returns the survey and all forms to the wallet, 

affixes a yellow dot to the wallet and files the wallet in the “Ready for Data Entry” 

file (Cabinet C).

Coding

In the coding process, certain items in surveys are marked to ensure data entry operators 

input the information correctly. All markings done in the surveys must be done in colored pen to 

distinguish ERG markings from respondent markings. A yellow highlighter pen will be used to 

highlight any information relating to CBI claims. Several markings or corrections should be done 

at this time:

• Check to see if the CBI box on Page iii is checked, indicating the entire survey is 

CBI. If this is the case, on page B-1, above the Part B header, write in “Data 

Entry, check global CBI box.” Highlight this text in yellow highlighter. 

• If the CBI box is not checked: 

1. Check any cover letter to determine if the respondent has claimed the 

entire survey (or Part B or any particular question in Part B) is CBI. If 

the respondent claims the entire survey, or all of Part B, as 

confidential, write in “Data Entry, check global CBI box” on page B-1 

as above. Highlight this text in yellow highlighter. 

2. If the cover letter indicates specific questions are confidential, make 

sure the box next to those questions have been checked “CBI.” If not, 

check the box, highlight it, and add “marked CBI as per cover letter,” 
QAPP—CBM Detailed Questionnaire

A-9

in the Reviewer Flag/Comment Sheet, with the relevant question 

number.

3. Check the comments section to determine if the respondent has 

declared any questions CBI, even if they are not marked as such. If 

questions have been claimed CBI in the comments section, but not 

marked as CBI in the check boxes, fill in the check boxes for the 

respondent and highlight them. Note this change in the Reviewer 

Flag/Comment Sheet.

4. Check the margins of the survey to see if the respondent has 

handwritten in CBI, confidential, do not disclose, etc. and fill in the 

check boxes for the respondent and highlight them for any question so 

designated, if not already checked. Note this change in the Reviewer 

Flag/Comment Sheet. 

5. Review each CBI box in Part B of the survey and if any CBI box for 

an individual question is checked (whether by the respondent or by 

ERG), highlight the box.

Following the CBI check, continue with the following general instructions and the 

question-by-question instructions listed below. Note that in general, -888 is used to designate 

questions that are blank because they are not applicable to the respondent and is also used to 

replace “NA” or “not applicable” answers made by respondents. Similarly, the code -999 is used 

for blanks where an answer was expected but was not found. The code -999 should not be used 

to replace any data. Another code, -777, is used to replace “unknown.” It is also used for any 

data that cannot be deduced from financial forms submitted for transcription data (e.g., revenues 

attributable to oil and gas production). Any changes to the answers other than those represented 

by these codes should be noted in the Reviewer Flag/Comment Sheet, unless otherwise noted. 

For example, if a box is checked “no”, which would mean the next question should be skipped, 

but the respondent has provided information in the next question, the reviewer should change the 

box to “yes” and note in the Reviewer Flag/Comment Sheet that the “no” box seemed to have 

been checked in error and it was changed to “yes.” Other details of coding are provided below.

• Look at the entries for Question B3-1. Check the project name and number in 

Appendix B (the complete list with CBI project names can be found in File E, 

Drawer 1). If the name checks, replace the name with the number. Write the 

number at the top of each column in the blank provided. If the respondent has 

written in the project name there, cross out the name and replace with the number.

• If the respondent reports fewer than 3 projects on Tables B-1, B-2, or B-3, strike 

through the columns not used using X’s in each column.

• If this respondent is not required to answer anything in Table B-2, place a large X 

through all pages of that table.
QAPP—CBM Detailed Questionnaire

A-10

• Respondents might use dashes instead of 0s. Strike dashes and replace with 0s. 

This does NOT require a notation in the Reviewer Flag/Comment Sheet.

• If respondents have filled in an answer for an “other” question but have not X’d 

the “other” box, X the box and add this to the Reviewer Flag/Comment sheet.

• Respondents might have used parentheses in place of negative signs in financials. 

Strike the parentheses and add a negative sign “-”.This does NOT require a 

notation in the Reviewer Flag/Comment Sheet.

• If a respondent has written a comment in the margin of any question, circle the 

comment, write COM next to the comment and add the question number next to 

COM, if question number is not clear. 

• If a respondent provides a range, write the midpoint of the range as the response. 

Circle the range given by the respondent and write COM under the question 

number. Data entry will enter the midpoint. This does not require a notation in the 

Reviewer Flag/Comment Sheet.

• If the respondent writes in “unknown” for any question, circle “unknown” and 

write COM under the question number, except as addressed in the specific 

instructions on questions here. Write in -777 as the answer for “unknown,” where 

“unknown” is not an option for an answer. Also use the code -777 to replace 

“unknown” as an answer to Question B2-2 if the respondent has checked “joint 

venture” in Question B2-1 or if they have responded to Question B3-2 with 

“unknown” (which also should be replaced with -777). 

Specific instructions on questions:

• If entries for B1-2 are missing, X out entire question.

• If entry for B1-3 is missing, X out entire question.

• If B2-1 Box 1 is checked, B2-2 and B2-3 should be blank or NA, and should be 

marked -888.

• If Box 2 in B2-1 is checked, and only one name appears in B2-2, -999 should be 

entered for name 2. Names 2 and 3 are -888 for all other respondents.

• If Box 2 in B2-1 is checked, and only one name appears in B2-3, -999 should be 

entered for name 2. However, if “unknown” appears, this entry can remain. Write 

in unknown for Names 2 and 3 as well. Names 2 and 3 are -888 for all other 

respondents.
QAPP—CBM Detailed Questionnaire

A-11

• If B2-10 is answered “yes” draw a line after the end of this question and write a 

note as follows: “Data Entry: you are done with Section B2. Turn to Page B-10.” 

X out Question B2-11 and Pages B-5 through B-9.

• If answers to questions B2-11 through B2-13 are missing, note -999. If B2-12 

through B2-14 have been marked “disagree,” but no alternative information has 

been provided in either the comments or in the underscored areas, circle 

“disagree” box and note COM, then check the “agree.” Box.

• If Question B-15 has been checked “Yes,” complete the coding of this question, 

but then skip to Question B2-23.

• In Question B2-19b and b2-19c: Respondent marks interest payments and or taxes 

as “unknown.” Strike out text and mark with a -888.

• Question B3-5 through B3-8: If respondent has answered “yes” to Question B3-4 

for a project, enter -999 if any of these questions has missing answers; otherwise 

use -888.

• If B3-9 is blank and B3-10 is blank, mark -888 in both questions. If either one is 

blank and the other is not, mark the blank question with -999.

• Question B3-11. Blanks in vertical should be marked -999. Blanks in horizontal 

should be marked -888.

• Questions B3-12 through B3-14. Mark all blanks as -888. 

• If both lines in B3-15 are missing, mark both -999. If one line or the other is 

missing, mark the missing line -888.

• Questions B3-18 and B3-19, mark -888 if B3-17 is marked “no”, otherwise mark 

-999.

• If B3-35 has an entry for either workovers, stimulations or recompletions, mark -

999 in B3-36 for any missing item with an expenditure marked in B3-35. For any 

item marked “NA” in B3-35, mark -888 in the matching item in B3-36. Otherwise 

mark any missing items as -999 in both questions.

• If Question B3-37 is answered yes or is missing (-999), and Questions B3-38 

through B3-40 are not answered, mark those questions -999, otherwise mark -888.

• If Question B3-38 is answered, B3-39 and B3-40 should have an entry for every 

line for which B3-38 has an answer; if not, missing answers should be marked -

999. Lines not used should be marked -888. NOTE: Cross out $ signs and 0s on 

Question B3-39.
QAPP—CBM Detailed Questionnaire

A-12

• If Question B3-41 is answered yes or is missing (-999), and Questions B3-42 

through B3-44 are not answered, mark those questions -999, otherwise mark -888. 

• If Question B3-45 is answered (including with a comment such as “unknown”) 

and Question B3-46 is not answered, mark Question B3-46 with -888. If Question 

B3-46 is answered with more than 1 lease filled in at least one of the answer 

options, mark Question B3-45 -888. If Question B3-46 shows only one lease, 

mark B3-46 -888 and transfer the lease type shown in B3-46 to B3-45. If both 

Question B3-45 and B3-46 are missing answers, mark both with -999.

• If Question B3-51 is answered with the 5% check mark, cross out the 5% check 

mark and write 5 in the “we estimate” line. If both are missing, write 5% in the 

“we estimate” line. Write “missing” in the margin, circle it and put it in the 

Reviewer Flag/Comment Sheet.

• If Question B3-53 is answered yes or is missing (-999), and Questions B3-54 

through B3-55 are not answered, mark those questions -999, otherwise mark -888. 

• If Question B3-56 is answered yes or is missing (-999), and Questions B3-57 

through B3-58 are not answered, mark those questions -999, otherwise mark -888. 

• If Question B3-68 is answered “yes” or is missing and B3-69 through B3-72 are 

not answered, mark B3-69 through B3-72 as -999. If B3-68 is answered “no,” 

mark B3-69 through B-70 as -888 and B3-71 through B3-72 as -888 if these latter

have not been answered. Replace any “NAs” with -888.

• If B3-68 has an answer and units or estimation method lines are missing requested 

information, mark -999, otherwise, mark units and estimation method -888.

• If Questions B3-78 through B3-85 are missing answers, check Table B-4 to see if 

the relevant information has been included there for at least 3 years. If not, check 

the box “we agree to this approach” and note that a question is missing an answer 

so the agree box was checked in the Reviewer Flag/Comment Sheet. If Table B-4 

contains the information, check the box saying projections have been provided in 

Table B-4 and note this in the Reviewer Flag/Comment Sheet.

• If the respondent has provided no information in Table B-4, X out all of Table B4 and note at the top of the Table “Data Entry—Skip to page B-37” (comments). 

If some information is provided, all missing items in Table 4 should be marked -

888. If entire years are missing, X out the columns.

Other issues may arise during the course of the coding effort. Consult with the ERG Task 

Manager (Anne Jones) for a decision; a decision will be written up and distributed to all coders 

for future reference. NOTE: The Reviewer Flag/Comment Sheet should ultimately reflect any 

coding in which answers have been transcribed (other than financial information transcribed in 

the financial transcription process. 
QAPP—CBM Detailed Questionnaire

A-13

Initial Problem Callbacks

The Detailed Reviewer 1 assigned the survey is responsible for all initial problem 

callbacks for that survey. The following steps should be followed for initial callbacks found in 

the “Initial Problems” file:

1. Pull an assigned survey from the “Initial Problems” file.

2. Follow steps 2 through 7 for Detailed Reviewer 1 (see above). (Note: Missing 

Part B, goes straight to callback).

3. Ensures Reviewer Flag/Comment Sheet is complete and prepares questions on the 

Callback Log Sheet (Form 7, Appendix C). If more than one sheet is needed, 

make sure sheets are stapled together either before or after the callback 

completion. 

4. Make the callback and note any answers provided in the callback log.

5. When missing information is provided following callback, transcribe information, 

or include new questionnaire sheets, noting in the callback log that new sheets 

have been provided and including date (e.g., Part B, relevant project sheets).

6. Enter survey into the review process at differing phases, depending on what 

information was missing:

a. Missing Part B: Returns to the active file for Initial Review; the Data 

Manager is informed that it is ready for Initial Review again.

b. Missing a project’s data: Receives a blue sticker and returns to the Active 

File for the completion of Detailed Review 1 (any completed portions of 

the Detailed Review 1 can be skipped). 

c. Missing financial reports: receives a blue sticker and is placed in the 

transcription file area and follows steps as outlined for surveys needing 

transcription from that point forward.

The following steps should be followed for initial callbacks found in the “Detailed 

Review Problems” File:

1. Pulls an assigned survey from the Detailed Review Problems file.

2. Reviews the Reviewer Flag/Comment Sheet and transcribe questions to the Callback Log 

Sheet (Form 7). If more than one sheet is needed, make sure sheets are stapled together 

either before or after the callback completion. 

3. Makes the callback and documents any answers provided on the callback log sheet.
QAPP—CBM Detailed Questionnaire

A-14

4. When all critical data is received (Form 2 data—either documented in the callback or 

provided in hardcopy form by the respondent), transcribes it into the questionnaire.

5. Makes additional callbacks if Form 2 data is still missing, until provided, or until 

instructed otherwise.

6. Reviews the completeness checklist and updates the list with the new information in 

place. Affixes a silver star, indicating first review is complete and/or proceeds directly to 

coding. 

7. Steps in Detailed Review 1 from 12d on can be continued from that point forward.

Data Entry

When the second review is completed, the reviewer has affixed a yellow dot and has 

placed the file in the data entry file drawer (File C, Drawer 3). The data entry personnel will 

complete all data and follow the instructions provided by the coding. At the completion of each 

form, the data entry personnel will make a quick QA check of their data entry information. At 

the completion of data entry, the file is returned to the data entry complete file, File C, Drawer 

1), with a gold star attached to show the data entry process is complete. 

At the end of the data entry process, a 20 percent check sample (based on numbers of 

projects) will be drawn and a complete proofreading of the sample will be done to gauge the 

accuracy of the data entry. If the quality check determines that the data entry is accurate, no 

further proofreading will be done. If accuracy issues are noted for any of the data entry 

personnel, all additional surveys entered by that person will be proofed. Corrections will be made 

directly to the database and proofed again.

Post-Data Entry Callbacks

These callbacks follow the same form as the initial callbacks, except that questions may 

be added by the ERG Task Manager following analysis of the database to identify out of range 

responses, totals that do not add, etc. These additional questions need to be added in with any 

questions derived from the Reviewer Flag/Comment sheet if an initial callback was not 

performed.

Color codes for file folders

Blue dot—Initial review complete or initial review data problem resolved, ready for detailed 

review.

Green dot—Financials require transcription.

Red star—First review started, needs initial callback.

Silver star—First review completed.

Red dot—First review and coding completed. Ready for second review. 

Yellow dot—Second review completed. Ready for data entry. 

Gold star—data entry completed.
QAPP—CBM Detailed Questionnaire

A-15

Figure 1. Survey Tracking Form

Survey ID Company Survey Receipt (date)

Folder 

Prepared 

(date)

Financial Info 

to be 

Transcribed

Financial 

Info 

Missing - 

CALLBACK

Completion of B2 

(date or 

NA)

Review & 

Coding 

#1 Complete 

(date)

Review & 

Coding 

#2 Complete 

(date)

Initial 

Callback 

Requir-ed 

(X)

Initial 

Callback 

Completed (date)

Data Entry 

Completed 

(date)

Database 

Data 

Checks 

Completed 

(date)

OID001

OID002

OID003

OID004

OID005

OID006

OID007

OID008

OID009

OID010

OID011

OID012

OID013

OID014

OID015

OID016

OID017

OID018

OID019

OID020

OID021

OID022

OID023

OID024
QAPP—CBM Detailed Questionnaire

A-16

APPENDIX A.1

REVIEW PACKET

(LOCATED IN EACH WALLET)
QAPP—CBM Detailed Questionnaire

A-17

2009 Coalbed Methane Questionnaire: Part B: Financial and Economic 

Information

REVIEWER COVER SHEET

7.1.1.1 Survey ID OID_____

Financial 

Reviewer 1

Financial 

Reviewer 2

Detailed 

Review 

Reviewer 1

Coder (if 

different from 

Reviewer 1)

Detailed 

Review 

Reviewer 2

Name

Date Started

Financials 

Entered/Chec

ked (Date)

Coding & 

Review 

Completed 

(Date)
QAPP—CBM Detailed Questionnaire

A-18

PART B1&B2

2009 Coalbed Methane Questionnaire: Part B1 &2: Financial and Economic Information

Overall Survey Completeness Checklist

Survey ID_________

Q# Yes No NA

Unreadable

Comment 

Explaining 

is Provided Checklist Item

Section B1

B1-1 Primary Contact Name is present

Contact Title is present

Email Address is present

Phone number is present

Fax number is present

Convenient time to call is present

B1-2 Primary Contact Name is present

Contact Title is present

Email Address is present

Phone number is present

Fax number is present

Convenient time to call is present

B1-3 B1-3 is present (if not present mark “NA”).
QAPP—CBM Detailed Questionnaire

A-19

Survey ID_________

Q# Yes No NA

Unreadable

Comment 

Explaining 

is Provided Checklist Item

Section B2

B2-1 One and only one box is checked

B2-2 Name of owner is present unless box 1 checked in B2-1, two or more owners 

present if box 2 checked in B2-1.

B2-3 Name of owner is present unless box 1 checked in B2-1, two or more owners 

present if box 2 checked in B2-1 (or “unknown” has been entered).

B2-4 One and only one box is checked

B2-5 Either a 6-digit number or an “NA” is entered in the underscored section

B2-6 One and only one box is checked

B2-7 One and only one box is checked; If “other” is a selection, an explanation has 

been entered

B2-8 One and only one box is checked

B2-9 Both underscored lines have a number inserted

B2-10 One and only one box is checked

For Respondents not answering “yes” to Question B2-10

B2-11 One and only one box is checked; If “other” is a selection, an explanation has 

been entered

B2-12 One and only one box is checked; if disagree, comment explaining is provided
QAPP—CBM Detailed Questionnaire

A-20

Survey ID_________

Q# Yes No NA

Unreadable

Comment 

Explaining 

is Provided Checklist Item

B2-13 One and only one box is checked; if “disagree” is selected, a number is entered 

on the underscored line or a number or other answer is provided in the 

comments section

One and only one box is checked, if “yes”, numbers are provided on all three 

underscores provided.

For Respondents not answering “yes to Question B2-15

B2-16 One and only one box is checked; If “other” is a selection, an explanation has 

been entered

B2-17 

to B2-

19

At least one number is entered each underscored section in each column of each 

table or an “NA” or “unknown” is written in.

B2-20 One and only one box is checked; If “other” is a selection, an explanation has 

been entered

B2-21/ 

B2-22

An entry has been made in each underscored section in each column of each 

table 

For All Respondents (other than those answering yes to Question B2-10)

B2-23/ 

B2-24

A entry has been made on each underscored line

B2-25 Item is checked if answer to B2-15 is yes and financials are attached.
QAPP—CBM Detailed Questionnaire

A-21

PART B3

2009 Coalbed Methane Questionnaire: PART B3: Project Information

Overall Survey Completeness Checklist

Survey ID_________ Project Name ______________________________________ Project ID ________________

Q# Yes No NA

Unreadable

Comment 

Explaining is 

Provided Checklist Item

B3-1/ 

B3-2

An entry has been made in the space provided

B3-4 One and only one box is checked

B3-5 to 

B3-8

An entry has been made in each of the spaces or underscores provided, 

if relevant (B3-4 = yes)

B3-9 Should have an entry if B3-10 has an entry

B3-10 Should have an entry if B3-9 has an entry

B3-11 At least one entry should be made for either vertical or horizontal

B3-12 to 

B3-14

An entry has been made in each underscored area

B3-15 A minimum of one entry has been provided in the underscored areas

B3-16 One and only one box is checked

Table B2—Only Respondents Marking “No” for Question B3-16) for a particular project

B3-17 One and only one box is checked

B3-18 to 

B3-36

An entry has been made in each of the spaces or underscores provided, 

if relevant

B3-37 One and only one box is checked
QAPP—CBM Detailed Questionnaire

A-22

Survey ID_________ Project Name ______________________________________ Project ID ________________

Q# Yes No NA

Unreadable

Comment 

Explaining is 

Provided Checklist Item

B3-38 to 

B3-40

An entry has been made in each of the spaces or underscores provided, 

if relevant

B3-41 One and only one box is checked

B3-42 to 

B3-44

An entry has been made in each of the spaces or underscores provided, 

if relevant

Table B3—All Respondents, All Projects

B3-45 to 

B3-50

An entry has been made in each of the spaces or underscores provided, 

if relevant

B3-51 One and only one box is checked

B3-52 An entry has been made each of the spaces or underscores provided, if 

relevant

B3-53 One and only one box is checked

B3-54 to 

B3-55

An entry has been made each of the spaces or underscores provided, if 

relevant

B3-56 One and only one box is checked

B3-57 to 

B3-67

An entry has been made each of the spaces or underscores provided, if 

relevant

B3-68 One and only one box is checked

B3-69 Units have been provided, one and only one box is checked for 

estimation method, if “other” has been selected, an entry has been 

made in the underscore provided, and a entry has been made in the 

space provided in the answer column(s)
QAPP—CBM Detailed Questionnaire

A-23

Survey ID_________ Project Name ______________________________________ Project ID ________________

Q# Yes No NA

Unreadable

Comment 

Explaining is 

Provided Checklist Item

B3-70 to 

B3-74

An entry has been made each of the spaces or underscores provided, if 

relevant; Units are specified, if relevant

Future Costs and Revenues (All Respondents, All Projects)

B3-75 One and only one box has been checked

B3-76, 

B3-77

An entry has been made in each of the spaces or underscores provided.

B3-78 One and only one box has been checked, underscores associated with a 

checked box have an entry

B3-79 to 

B3-85

One and only one box has been checked, underscores associated with a 

checked box have an entry, if applicable

B3-86 Optional; if provided make sure it’s readable

For respondents who did not check the first box of Question B3-75 (remember to consider NAs if boxes are checked in 

Questions B3-76 through B3-85)

B3-87 to 

B3-102

An entry has been made each of the spaces or underscores provided, if 

relevant

B3-103 Option; if provided made sure this is readable
QAPP—CBM Detailed Questionnaire

A-24

APPENDIX A.2

CBM FORMS
QAPP—CBM Detailed Questionnaire

A-25

Survey ID____

FORM 1

INITIAL REVIEW

Issue Yes No

1. Out of Scope

2. Missing All of Part B

3. Financial Data for B2 Missing (No Financials)

4. B2 Needs Transcribing

5. Projects Missing

6. Gas & Water Projections Are Missing (Respondent 

Indicated Attachment)

Note: 

Out of Scope marked “Yes” goes in File E, Second Drawer

Issues 2, 3, or 5 marked “Yes” goes in File D, Fourth Drawer

If Issue 4 is marked yes (but Issues 2, 3, or 5 are not), survey goes in File C, 

First Drawer.

Issue 6 gets logged on Form 3 and placed back in active file with blue sticker

All Others Place Back in Active File with Blue Sticker

Reviewer Initials _________

Date____________________
QAPP—CBM Detailed Questionnaire

A-26

FORM 2

INITIAL CALLBACK TRIGGER FORM

Issues Yes No

1. B2-1, B2-4, or B2-9 are missing

2. B2-17e, B2-18i, B2-19a, B2-19d, B2-21d, B2-22c or B2-22d are 

missing for all years

3. B3-1 (if responding for more than 1 project) is missing

4. B3-4 = yes and B3-5 through B3-8 are missing

5. B3-11 or B3-15 are missing

6. B3-16 = no and Table B-2 is missing all information

7. B3-16 = no and B3-20 through B3-24 are missing (any one of 

these)

8. B3-16 = no and B3-26 through B3-29 or B3-34 are missing 

(any one of these)

9. B3-48 >0 or not blank and B3-47 is missing

10. B3-50, B3-59 or B3-62 through B3-66 are missing (any one of 

these)

11. B3-76 or B3-79 are missing and respondent has not filled out 

at least 3 years of Table B-4

12. Too many missing or illegible answers

Note: If any issues are marked “Yes” place in File C, Fourth Drawer.
QAPP—CBM Detailed Questionnaire

A-27

FORM 3

SUMMARY OF INITIAL NOTES AND PROBLEMS FOR PART B

(Mark X Only for Survey IDs with Items Missing or Noted)

Survey 

ID

Out of 

Scope

Missing 

Part B

Missing Financial 

Reports

Missing 

Project(s)

Projection 

Attachments

Missing

Projections 

Attached
QAPP—CBM Detailed Questionnaire

Form 4

List of Surveys in Transcription

(Place Reviewer Initials and Date in Appropriate Column)

Survey ID

Transcription 

Begun

Transcription 

Completed

Review of 

Transcription 

Begun

Review of 

Transcription 

Completed
QAPP—CBM Detailed Questionnaire

Form 5

List of Surveys in Detailed Review & Coding

(Place Reviewer Initials and Date in Appropriate Column)

Survey 

ID

Review 

1 Begun

Review 1 

Completed

Coding 

Begun

Coding 

Completed

Review 

2 Begun

Review 2 

Completed
QAPP—CBM Detailed Questionnaire

FORM 6

2009 Coalbed Methane Questionnaire: Part B: Financial and Economic 

Information

REVIEWER FLAG/COMMENT SHEET

7.1.1.2 Survey ID ______________________

Company Name ______________________

Reviewer 1 ______________________

Reviewer 2 ______________________

Question 

No. Description of Problem

Date Resolved & 

Reviewer’s Initials 
QAPP—CBM Detailed Questionnaire

FORM 7

2009 Coalbed Methane Questionnaire: Part B: Financial and Economic 

Information

Project Name: CBM Detailed Study

Project No: 

Facility ID: 

Facility Name: 

Contact Name: Contact Phone No: Contact Email Address: 

Date: Staff Name: 

Time: AM PM

This document contains 

CBI: 

_____ Yes

_____ No

Type of Contact:

 Email Telephone 

General Subject(s) 

Discussed: 

CBI =

Yes (Y)

Question

No. Topics Discussed