MEMORANDUM

TO: Project Record

FROM: EPA

DATE: 12 February 2013

SUBJECT: Quality Assurance Activities During the Review of the Economic Portion of 

the Detailed Questionnaire for the Coalbed Methane (CBM) Extraction Sector

and the Development of the Economic Database

EPA distributed the Detailed Questionnaire for the Coalbed Methane (CBM) Extraction 

Sector1 (Detailed Questionnaire) to 105 out of 251 CBM operators (U.S. EPA, 2009a). The

Detailed Questionnaire comprises the engineering portion (Parts A and C) and the economic 

portion (Part B). EPA received responses from 88 operators in hardcopy format, which EPA 

reviewed and compiled in several Detailed Questionnaire databases for subsequent industry 

analyses. Table 1 summarizes the information requested in the Detailed Questionnaire. 

Table 1. Summary of Information Requested in the Detailed Questionnaire for the 

CBM Extraction Sector

Part A: CBM Projects2

Section 1 – Project 

Identification

Selected for Questionnaire and Certification 

Identifies the projects and wells associated with the operator and determines 

whether the operator responded to the Screener survey3

Part B: Financial and Economic Information

. (U.S. EPA, 2009b)

Section 1 – Contact 

Information

Identifies a contact for the financial and economic portion of the 

questionnaire.

Section 2 – Firm-level 

Financial 

Includes firm-level financial information at the lowest corporate level (i.e., 

the immediate owner or operator). 

Section 3 – Project-level 

Questions

Includes project-specific financial information from the operator who 

manages the day-to-day operations of the project. 

Part C: Produced Water Management System (PWMS) Information

Section 1 – General 

Operator Information

Identifies a contact for the technical portion of the questionnaire. 

Section 2 – PWMS General Includes project-specific information on the components and costs of the 

produced water management system that is used for the project. 

 1 EPA used the information collected from the CBM industry in the Screener Questionnaires to select a sample of 

operators and projects required to complete the Detailed Questionnaire. The statistical sampling plan is located in 

the CBI docket. (Batelle, 2009) 2 EPA identified the primary unit of analysis for the CBM industry as a “project,” defined as a well, group of wells, 

lease, group of leases, or recognized unit that is operated as an economic unit when making production decisions.

3 The Screener requested operators to verify small business status; CBM operations in 2008; and number of projects 

operated. For each project, the Screener requested for calendar year 2008 the number of wells, gas production, and 

produced water management methods.
Memorandum

February 12, 2013

Page 2

Table 1. Summary of Information Requested in the Detailed Questionnaire for the 

CBM Extraction Sector

Section 3 – Detailed PWMS 

and Treatment 

Includes project-specific information related to the discharge and treatment 

practices of the produced water management system. 

Section 4 – Produced Water 

Quality Data

Includes water quality data submitted by the operator, which may include 

data for untreated produced water, treated produced water, and outfall 

discharges.

This memorandum describes the review and database development activities for the

economic portion of the Detailed Questionnaire (Part B)

4

. ERG, EPA’s contractor for the 

economic portion of the CBM rulemaking, conducted the questionnaire review and database 

development according to the Quality Assurance Project Plan for the Economic Portion of the 

Coalbed Methane Detailed Questionnaires and Coalbed Methane Economic Analysis Task 

(Economic QAPP) approved on March 21, 2012 (U.S. EPA, 2012). ERG received a majority of 

the hardcopies of the Detailed Questionnaire from EPA in March 2010 and the remainder of the 

hardcopies in May 2011. EPA had completed some initial follow-up with operators upon receipt 

of the questionnaires; however, upon EPA direction, ERG did not complete follow-up activities. 

This memorandum discusses the steps ERG took to review the questionnaires and develop a 

database. 

Section 1 discusses the questionnaire review procedures, Section 2 discusses the 

development of the economic database, Section 3 discusses additional updates to the data, and 

Section 4 discusses the data quality of the Detailed Questionnaire responses and database.

1.0 REVIEW OF THE ECONOMIC PORTION OF THE DETAILED QUESTIONNAIRE (PART B)

Table 2 summarizes EPA’s distribution and receipt of the Detailed Questionnaire 

responses. Table 3 summarizes the quality control (QC) activities ERG performed during the 

review. The following subsections describe the specific procedures that ERG used to ensure 

quality during review or the economic portion of the Detailed Questionnaire. 

 4 The review of the engineering portion of the Detailed Questionnaire is described in a separate memorandum, 

Quality Assurance Activities During the Review of the Engineering Portion of the Detailed Questionnaire for the 

Coalbed Methane (CBM) Extraction Sector and the Development of the Detailed Questionnaire Databases. (U.S. 

EPA, 2013a)
Memorandum

February 12, 2013

Page 3

Table 2. Summary of Responses to the Economic Portion of the 

Detailed Questionnaire

Number of Detailed 

Questionnaires 

 Distributed to Industry 114

 Indicated Out of Scope – No CBM wells a 7

 Partial or Non-Respondent b 19

 Economic Portion (Part B) Reviewed 88

a – The count of out-of-scope questionnaires only includes those questionnaires 

that indicated that they do not operate any CBM projects. 

b – The count of partial or non-respondent questionnaires only includes those 

questionnaires that did not respond or only responded partially to all projects for 

which EPA requested information. Partial questionnaires were not reviewed. 

Table 3. QC Statistics for Review and Data Entry of the Detailed Questionnaire Responses

Number of Detailed 

Questionnaires

Percent Reviewed

Initial Technical Review 88 100%

Second Technical Review a 88 100%

Data Entry into Economic Database 88 100%

Data Entry First Review b 88 100% c

Data Entry Second Review 80 91% d

a – Second technical review of the Detailed Questionnaire was conducted by a senior staff member familiar with 

the CBM Detailed Questionnaire, the CBM industry, and the issues operators had completing the questionnaire. 

b – Data entry review was conducted by a technical staff member familiar with the technical review (either as 

initial reviewer or second reviewer).

c – ERG received responses for one Detailed Questionnaire in 2012 and reviewed and data entered the responses 

into the database at a later time (with data entry first review). This was not part of the original 2010 review 

process; however, ERG implemented the same review procedures for this questionnaire. 

d – ERG data entered and second reviewed 80 of the Detailed Questionnaires in 2010 prior to the end of the 

contract period. During the 2011 review, ERG determined the data entry of the responses provided a high level 

of accuracy and did not continue the data entry second review of the remaining questionnaires.

1.1

ERG developed an internal tracking spreadsheet (Figure 1 of Economic QAPP, Appendix 

A) to quickly identify issues with Part B of the Detailed Questionnaire and to keep track of the 

receipt, review, and transcription of the responses to Part B of the questionnaire. Key fields in 

the tracking database table include:

Log-in and Tracking Procedures

• Survey ID

• Company

• Survey Receipt (Date)

• Folder Prepared (Date)

• Form One Complete

• Financial Transcription Required
Memorandum

February 12, 2013

Page 4

• Financial Transcription Complete

• Review 1 and Coding Complete 

• Review 2 Complete

• Callback Required

• Callback Complete

• Data Entry Complete

• Initial Checks of Data Entry

• Review of Data Entry Checks

Each Detailed Questionnaire received by ERG was stored in a designated folder with a 

review packet (Appendix A.1 of the Economic QAPP) to document the levels of review for each 

questionnaire. ERG maintained separate file drawers for responses in each step of the review 

process. As one step was completed for a questionnaire, it was placed in the designated file 

drawer for the next step to further identify the questionnaires by stage of the process. 

Additionally, at the completion of a step, each folder was marked with a colored dot, which 

signified what stages the questionnaire had already passed through. Thus, any out-of-place 

folders could be immediately recognized. As each questionnaire passed through each stage, the 

staff person who performed the specific task (e.g., First Reviewer) initialed a form appropriate to 

the stage of the review and coding process (held within the questionnaire folder), and reported 

the completed stage to the review and coding supervisor, who updated the tracking spreadsheet.

EPA gave respondents the option to claim answers to specific questions as CBI by 

marking question-specific indicators in the Detailed Questionnaire or to claim all eligible data 

CBI by marking the “allCBI” box at the beginning of the Detailed Questionnaire. During the 

review of the Detailed Questionnaire, ERG handled all questionnaires as CBI, as specified in the 

Economic QAPP.

1.2

ERG reviewed the responses to the economic portion of the Detailed Questionnaires. 

Review and Coding of Questionnaire Responses

ERG had conducted some initial follow-up during the 2010 review of the questionnaires. When 

ERG continued review in 2011 under a separate contract and under EPA direction, we did not 

conduct any additional follow-up. 

An initial review determined whether all key sections and questions in the Detailed 

Questionnaire were completed. Some Detailed Questionnaires were largely blank and were 

counted as non-responses. EPA offered to transcribe financial data for small operators if they 

included their tax forms and financial reports with the Detailed Questionnaire. During the initial 

review, ERG determined whether financial transcription was necessary for each response (see 

section 2.0 for further details). 

After completing the financial transcriptions, ERG performed the first detailed review of 

each questionnaire and organized respondent comments. To preserve the original hardcopies of 

the questionnaires, ERG created and marked up review copies of the Detailed Questionnaires 

according to the Instructions for Survey Tracking, Review and Coding (Appendix A in the 
Memorandum

February 12, 2013

Page 5

Economic QAPP). ERG reviewed the completeness and reasonableness of the questions and 

flagged discrepancies and documented any corrections in Form 6 of Appendix A of the 

Economic QAPP. 

ERG conducted a second detailed review of all the Detailed Questionnaires. ERG 

documented the issues found during second review on Form 6 but did not complete any followup calls, as directed by EPA. The initial technical reviewer resolved any issues from second 

review and noted additional changes to the questionnaire on the review copy, Form 6, and the 

respondent comment section of Part B. 

ERG also coded the responses during review to ensure they could easily be entered into a 

database. ERG used the following codes during review:

• Unknown (respondent wrote in unknown): -777 

• Not applicable: -888

• Responded left a required response blank: -999.

1.3

This section describes ERG’s data entry and proofing procedures. 

Data Entry and Database Development

1.3.1 Data Entry and Initial Checks

ERG developed the economic database to house the questionnaire responses by 

constructing tables that followed the design of the Detailed Questionnaire. The CBM economic 

survey database is a confidential database that is located on EPA’s CBI docket (U.S. EPA, 

2013b). The memorandum, Summary of Confidential Business Information Claims in Industry 

Responses to EPA’s CBM Screener Surveys and Detailed Questionnaires and Industry Data 

Available in the EPA Docket, describes EPA’s decision to maintain this database in the CBI 

docket (U.S. EPA, 2013c). The database dictionary provides a description of the tables and data 

fields included in the economic database (U.S. EPA, 2013d).

Because EPA received most of the responses to the Detailed Questionnaires in hardcopy, 

ERG manually entered all responses into the database, including all corrections, coding, and 

documented changes. Using the review copies of the Detailed Questionnaires, ERG constructed 

the economic database and transcribed the responses or ERG’s corrections to the responses into 

the economic database. ERG performed first review of 100% and second review of 90% of the 

questionnaires entered into the database by visually comparing all the information contained in 

the database to the review copy. ERG internally tracked the data entry and data entry review in 

the survey tracking form. 

After data entry was completed, ERG performed a series of quality control and assurance

checks, including queries to check reporting consistency between questions and to check 

responses against publicly available data. See Section 4.0 for details of ERG’s quality control 

procedures. 
Memorandum

February 12, 2013

Page 6

1.3.2 Data Entry Proofing

A reviewer (a person who had not entered the data) performed the first data entry review 

for 87 of the 88 responses (99%)5 by visually comparing all the information contained in the 

database to the review copy. A second reviewer reviewed the changes that were made in the first 

data entry review for 80 of the 88 responses (90%) against the review copy. ERG documented 

these reviews on Forms 4 and 5 in Appendix A of the Economic QAPP. 

After ERG completed data entry, data checks, and data entry review of the Detailed 

Questionnaires, ERG performed database clean-up and standardization activities as needed to 

facilitate subsequent data analyses. Sections 3.0 and 4.0 describe these activities.

2.0 EPA REVIEW AND PROCESSING OF DETAILED QUESTIONNAIRE ATTACHMENTS FOR 

PART B

CBM operators may have included the following supplemental information in 

attachments to Part B of their Detailed Questionnaires:

• Tax forms (generally, S-Corp filings);

• Financial reports (income statements and balance sheets); and

• Production data, decline curves, or gas and water projections.

For respondents requesting assistance with their financial information, ERG identified 

relevant financial data in financial reports or tax forms and transcribed the data into Part B-2 of 

the Detailed Questionnaire. ERG transcribed financial information for 15 questionnaires. 

Because inclusion of financial reports or tax forms was only required from those requesting 

assistance, ERG was not able to fill in missing or incorrectly entered or calculated financial 

information from most other questionnaire respondents. In a few cases, however, missing data or 

data that did not compute correctly could be checked against Security and Exchange 

Commission (SEC) submissions for publicly held firms. 

When ERG found data to complete the firm level information for such firms, ERG 

transcribed the new numbers into the Review copy and added the printout of the relevant pages 

from the annual report to the questionnaire folder. Respondents provided financial attachments in 

hard copies so ERG manually transcribed relevant information onto the Review Copies of the 

responses. Before transcription, however, ERG used data from the attached financial reports 

and/or tax forms to calculate the specific financial metrics requested in the Detailed 

Questionnaire. 

A senior staff member performed the calculations and initial transcription. An ERG 

analyst who did not perform the initial calculations and transcription reviewed the transcription 

 5 One Detailed Questionnaire was considered a non-response, because part of it was not received by ERG until after 

the quality control checks of the Economic Database were completed. After this response was entered into the 

database, the data entry was checked, though not as a part of the formal data entry and review effort.
Memorandum

February 12, 2013

Page 7

process verifying that all calculations were accurate and that the results had been correctly 

transcribed onto the review copies. If any changes appeared to be needed, the two analysts 

conferred to ensure that the correct numbers were transcribed. Any changes made were checked 

by both analysts. Only when a final pass was made by both analysts with no changes needed was 

the questionnaire returned to the review and coding process. 

If the respondent had provided gas and water production attachments in lieu of answering 

production questions, ERG performed calculations and transcriptions followed the same model 

as that for financial data. The Detailed Questionnaire requested that respondents provide decline 

curve and/or projection attachments. ERG did not transcribe these materials but retained them in 

the folder to assist in projection estimates.

3.0 ADDITIONAL UPDATES TO THE DETAILED QUESTIONNAIRE RESPONSES

ERG incorporated additional information from public data sources into the economic 

database. This section discusses how ERG obtained this information and added it to the Detailed 

Questionnaire responses contained in the economic database. 

3.1.1 Public Data Used to Fill in Missing Responses for the Detailed Questionnaire

Whenever a response appeared to be an outlier, or information provided in the response 

conflicted with information collected in the screener survey, ERG verified non-financial key data 

elements such as gas produced, sold, and used; water produced; firm-size; and acquisition and 

operating dates against publicly available databases such as operator websites, SEC filings, and 

state oil and gas commission websites. All corrections were documented on the Review Copies 

of the Survey, and in the comments section of Part B of the Detailed Questionnaire, which were 

data entered into the economic database. ERG downloaded copies of the information obtained 

from SEC filings that were used to correct questionnaire responses and included them in the 

operator file. 

3.1.2 Verifying Net Revenues for Discharging Projects

Based on 2008 revenues, costs, and other financial metrics, ERG estimated 2008 and 

2010 Net Revenues to analyze the impacts of potential costs. ERG used the Producer Price Index 

(PPI) for the Crude Petroleum and Natural Gas Extraction Industry to inflate financial data 

responses to the desired year and used the average over the questionnaire-reported years 2006-

2008 (in 2008 and 2010 dollars, respectively) to estimate firms’ and projects’ typical net 

revenues. ERG performed several additional data checks and modifications at this stage to ensure 

that gas production, water production, taxes, and other relevant estimates were reported with 

consistent units across firms. Also some respondents did not report key elements used to 

calculate net revenues, or their responses were well outside the range of expected responses. 

Where data were not reported or an extreme outlier was suspected, ERG imputed average 

wellhead prices or royalty, severance, and ad valorem tax rates based on the responses of others 

operating in the same basin. ERG described all changes and additions to respondent-provided 

information in the comments field of the economic database.
Memorandum

February 12, 2013

Page 8

4.0 DATA QUALITY OF THE DETAILED QUESTIONNAIRES AND ECONOMICS DATABASE

ERG performed extensive checks on the responses to Part B of the Detailed 

Questionnaire to verify that the financial information collected and the economic database 

containing the information was complete, accurate, and reasonable. This section documents

ERG’s review of the data quality of the detailed questionnaire responses and the economics 

database.

4.1.1 Completeness

ERG reviewed 88 out of 88 Detailed Questionnaires (100%). ERG used Instructions for 

Survey Tracking, Review and Coding (Appendix A of Economic QAPP) to review questionspecific checks for completeness and reasonableness. As directed by EPA, ERG did not continue

follow-up activities during its 2012 review of the Detailed Questionnaire responses. Subsequent 

industry analyses using the economic database included only the data available from the 

complete responses or from responses that ERG completed using available information 

throughout the questionnaire. 

Table 4 summarizes ERG’s check on completeness for questions that were used in EPA’s 

industry analyses, including responses to questions on revenues, taxes, costs, and gas production. 

Responses that were illegible, unknown or NA, or left blank (but required) were coded according 

to the instructions and not used in any subsequent EPA analyses.

Table 4. Completeness of Revenues, Costs, and Production Questions in the Detailed 

Questionnaire

Question 

Number Question Description

Number of 

Complete 

Responses

Number of 

Expected 

Responses

Percent 

Complete

B2-17e Total revenues 65 88 74

B3-50 Gas production operating costs 170 172 98

B3-59 Gas production 170 172 98

B3-62 Average project wellhead price 166 172 97

B3-65 Royalty rates 166 172 97

B3-66 Severance rates 166 172 97

B3-67 Other taxes 165 172 96

4.1.2 Reasonableness

ERG performed question-specific consistency checks on the financial data used in 

EPA’s industry analyses. In general, the majority of the data provided seemed reasonable given 

ERG’s knowledge of the individual operators, how they answered their screeners, and how the 

CBM industry operates. ERG performed the following to ensure we reviewed in more depth any 

potentially unreasonable data and made corrections, or imputed data, where possible: 
Memorandum

February 12, 2013

Page 9

• Checking for outliers, determining which could be possible, and which were far 

beyond any reasonableness using the screener, publicly available information, or 

general knowledge of usual conditions or circumstances in the industry (for 

example, knowledge that a 0.2 percent royalty rate is highly unlikely but that 

reporting .2 instead of a much more likely 20 percent might be a more reasonable 

circumstance); 

• Checking for summations where items were supposed to sum, checking for units 

and order of magnitude issues (reporting $ instead of $1,000s, e.g.); 

• Checking outliers against publicly available information; and

• Checking other information where general knowledge of the industry indicated 

the answers might have been in error against screener information or publicly 

available information.

ERG recorded all corrections on the Review Copy of the Detailed Questionnaire and 

updated the data entry for each response in the economic database.

4.1.3 Accuracy

ERG conducted review of the initial data entry for all questionnaires into the economic 

database and conducted a second review for 91% of the questionnaires after initial checks and 

proofing changes were made to the database. 

ERG also checked that responses reporting total revenues and costs for all CBM projects 

in 2008 in section B-3 of the Detailed Questionnaire matched those reported for CBM revenues 

reported by the firm in Section B-2. ERG ran the financial data queries iteratively to ensure that 

all errors that could be resolved had been resolved when fixes were made. Thus, ERG 

systematically verified the financial data for all discharging operators to identify any incorrect 

financial estimates reported as a part of the initial responses.
Memorandum

February 12, 2013

Page 10

5.0 REFERENCES 

1. Batelle, 2009. Coalbed Methane Study: Comparison of Original to Final Detailed 

Questionnaire Sample Design . Batelle Memorial Institute. May 27, 2010. EPA-HQ-OW2010-0824, DCN CBM00635.

2. U.S. EPA, 2009a. Detailed Questionnaire for the Coalbed Methane (CBM) Extraction 

Sector. U.S. Environmental Protection Agency. October 2009. EPA-HQ-OW-2006-0771, 

DCN 10072.

3. U.S. EPA, 2009b. Coalbed Methane Industry Screener. U.S. Environmental Protection 

Agency. 2009. EPA-HQ-OW-2006-0771, DCN 10068.

4. U.S. EPA, 2012. Quality Assurance Project Plan for the Review and Data Entry of the 

Economic Portion of the Coalbed Methane Questionnaire. U.S. Environmental Protection 

Agency. March 21, 2012. EPA-HQ-OW-2010-0824, DCN CBM00692.

5. U.S. EPA, 2013a. Quality Assurance Activities During the Review of the Engineering 

Portion of the Detailed Questionnaire for the Coalbed Methane (CBM) Extraction Sector 

and the Development of the Detailed Questionnaire Databases. U.S. Environmental 

Protection Agency. February 2013. EPA-HQ-OW-2010-0824, DCN CBM00663.

6. U.S. EPA, 2013b. CBM Economic Survey Database (Microsoft Access 2003 database).

U.S. Environmental Protection Agency. February 2013. EPA-HQ-OW-2010-0824, DCN 

CBM00656.

7. U.S. EPA, 2013c. Summary of Confidential Business Information Claims in Industry 

Responses to EPA’s CBM Screener Surveys and Detailed. U.S. Environmental Protection 

Agency. February 2013. EPA-HQ-OW-2010-0824, DCN CBM00661.

8. U.S. EPA, 2013d. CBM Economic Survey Database Dictionary. U.S. Environmental 

Protection Agency. February 2013. EPA-HQ-OW-2010-0824, DCN CBM00659.