Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

QA /QC OF INVEN TORY SYSTEMSACKNOLEDGEMENTS
This paper was written by Joe Mangino (ERG Incorporated).

QA/QC of Inventory Systems

507

Background Paper

1

INTRODUCTION

1.1

Background

The IPCC Guidelines for National Greenhouse Gas Inventories (IPCC Guidelines) describe steps for reporting
national inventories (IPCC, 1997). The third step, verification, contains a task labeled “Assessing Quality”. The IPCC
Guidelines call for a self-assessment of the quality of the inventory and provide a table for reporting quality ratings interms of “high”, “medium”, and “low” confidence. In order to properly perform and communicate this self-assessmentof quality, it is necessary for countries to implement quality assurance (QA) and quality control (QC) procedures as
integral parts of inventory development.
The goal of the IPCC good practice guidance is to develop emission inventories that “can be readily assessed in termsof quality and completeness.” Accomplishing this goal requires that minimum levels of QA/QC be implemented bycountries as they develop their emission inventories. This paper identifies and defines fundamental QA/QC
programme elements that should serve as the basis for good practice in every country’s emission estimation processes.
The objective of this paper and of the QA/QC work session at the Scoping Meeting on Managing Uncertainties will be
to develop good practice guidance on the recommendations and procedures for QA/QC, which can be directly
incorporated into the existing IPCC Guidelines. The approaches outlined in this paper should serve as guidelines inachieving this objective. The QA/QC good practices that are ultimately developed should reflect practicality,
acceptability, cost-effectiveness, existing experience, and potential for codification in order to be implemented
uniformly on a worldwide basis.
In the context of the United Nations Framework Convention on Climate Change (UNFCCC) Reporting Guidelines on
Annual Inventories (FCCC/SBSTA/1999/L.5), a QA/QC programme is an integral part of good practice in inventory
development. Elements of a QA/QC programme contribute to the objectives of good practice guidance, namely to
improve transparency, consistency, comparability, completeness, and accuracy. Also, the UNFCCC Reporting
Guidelines state that national inventory reports from countries should include “information on QA/QC procedures
implemented” (FCCC/SBSTA/1999/L.5). The procedures outlined in this report can provide information to support
these reporting guidelines

1.2

Definitions

The terms ‘quality control’ and ‘quality assurance’ are often used interchangeably or interpreted to mean different
things. The following definitions of QC and QA will be used for the purposes of updating the IPCC Guidelines andframing the recommendations for good practice:
Quality Control is a system of routine technical activities, implemented by inventory development personnel tomeasure and control the quality of the inventory as it is being developed. The QC system is designed to:•Provide routine and consistent checks and documentation points in the inventory development process toverify data integrity, correctness, and completeness;•

Identify and reduce errors and omissions;•

Maximize consistency within the inventory preparation and documentation process, and•Facilitate internal and external inventory review processes.

QC activities include technical reviews, accuracy checks, and the use of approved standardized procedures foremission calculations and measurements.
Quality Assurance activities include a planned system of review and audit procedures conducted by personnel not
actively involved in the inventory development process. The review should be performed by an independent, objective
third party to assess the effectiveness of the internal QC programme development, to verify that data quality objectives
were met, and to reduce or eliminate any inherent bias in the inventory processes.

1.3

Organization of guidance

There are three main steps for countries to follow in order to integrate QA/QC procedures as part of their inventory
development process:•508

Establish a written QA/QC plan;

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories•

Implement the QA/QC plan, and•Document and report the QA/QC activities.

In the following sections of the report, good practice recommendations are provided for each of the above steps inorder to successively implement and demonstrate an inventory quality assessment. In addition, supplementary QA/QC
activities are discussed or referenced in the report. These supplementary QA/QC activities are strongly encouraged andcan provide valuable information to more completely demonstrate and report on quality assessment.
In addition to this guidance, inventory preparers should refer to the specific QA/QC procedures being developed aspart of the good practice guidance for individual source categories. The source category guidance reports discuss
particularities associated with the QA/QC of individual source categories that are not contained in this document.

2

ELEMENTS OF A QA/QC PLAN

The first step in applying QA/QC procedures is to develop a structured plan of activities. It is recommended that a
written QA/QC plan be developed prior to any emissions being calculated or measured. The plan should outline the
QA/QC activities that will be implemented, along a scheduled time frame of implementation that follows inventory
preparation from its initial development to final reporting. The QA/QC plan should include the following components:•

Organizational structure;•

General inventory level QC procedures;•

Source-specific QC procedures, and•QA procedures

The written QA/QC plan should not represent an extensive new reporting requirement, but should be utilized as aninternal reference to organize, plan, and implement QA/QC activities. The QA/QC plan, once developed, can bereferenced and utilized in subsequent inventory developments, with slight modifications when QA/QC activities are
modified.
Good Practice Recommendation:
Each country should have a written QA/QC plan that describes the organizational structure for the
QA/QC programme and the QA/QC procedures that are implemented. This written plan should be
kept as internal documentation at the country level and be available for outside review if necessary.
The following sections provide guidelines on minimum and supplemental procedures for each of the above
components of the QA/QC plan.

2.1

Organizational structure

The organizational structure identifies the roles and responsibilities for QA/QC activities. Centralized coordination of
QA/QC activities is recommended for the successful implementation of a QA/QC programme. Centralized
coordination means that there should be an organized approach to QA/QC activities, with centralized oversight fromthe agency charged with preparing the inventory (hereafter called the “preparing agency” or the “inventory agency”) toensure that the QA/QC plan is implemented. QA/QC is a preparing agency function. Depending on the individual
circumstances for each country, activities related to QA/QC may be assigned to third parties or consultants, but theplan, integration, and review of these QA/QC activities need to have oversight from the preparing agency.
Good Practice Recommendation:
Assignments for QA/QC responsibility should cover all source categories included in the inventory.
If inventory assignments are made to outside agencies or consultants, the inventory preparing
agency should specify the minimum required QA/QC elements expected according to good practiceguidance and should review the QA/QC activities to ensure that they meet that expectation.
As supplemental guidance, it is highly encouraged that the inventory preparing agency identifies a QA/QC
coordinator, an individual who is responsible for ensuring the objectives of the QA/QC programme are implemented.
If at all possible, the QA/QC coordinator should be an individual who is not a member of the inventory staff that is
preparing the emission estimates. This organizational structure instills validity in the QA/QC activities and results

QA/QC of Inventory Systems509

Background Paper

since many of the activities rely on a truly objective analysis. Also, an independent QA/QC team can help minimize
bias in the overall inventory process.
Supplemental guidance on the use of the International Organization for Standardization (ISO) 9000 series for
implementing and organizing QA/QC activities is provided in Annex 1 of this report. The ISO 9000 series is aninternationally recognized data quality management programme that could be useful to planning and organizing
QA/QC activities for the entire inventory. Such formal data management systems can be adopted as part the written
QA/QC plan where appropriate.

2.2

General QC procedures

The written plan should describe the general QC procedures a country plans to undertake. General QC proceduresconsist of routine checks that are implemented across all phases of inventory development and do not necessitate
having specific knowledge of a given source or sink category. The focus of general QC techniques is on the processing,handling, and reporting procedures that are common to all the inventory sources. The other aspect of general QC
techniques is that they can be implemented by personnel that do not necessarily have specific knowledge of a particularsource category.
Table 1 lists the recommended minimum QC checks that preparing agencies should be routinely using throughout thedevelopment of the inventory. The preferred technique for most of the checks shown in Table 1 is to perform thechecks manually, by either recalculating by hand or through visual inspection of the records. Manual checks avoid thepossibility of introducing error into the QC check itself, which may be possible if only automated programmes are
utilized.
Good Practice Recommendation:
The minimum QC procedures listed in Table 1 of this report should be implemented for all source
categories. If parts of the inventory are prepared by outside agencies or consultants, the preparing
agency should review the general QC procedures implemented by the outside agencies or
consultants to ensure that they meet these minimum requirements.
The QC checks listed in Table 1 are directly applicable to inventory calculations and processing steps conducted by thecountry’s inventory preparing agency. However, in some cases, emission estimates are prepared for the country byoutside consultants or agencies. When making inventory assignments to outside consultants or agencies, countries
should communicate the need for the minimum QC checks listed in Table 1. This will ensure that the QC activities areaccomplished and recorded by the outside agency or consultant, and can be simply reviewed by the country’s
inventory staff to verify their implementation.
In some cases, it will not be feasible to run all the checks listed in Table 1 on every calculation and every parameterused in the inventory. In these cases, preparing agencies should choose selected subsets of data to perform the checks
on. The size of the subset will depend on the overall number of data points, records or calculations that have to bechecked, the complexity of the calculations, and the error rate encountered as QC progresses. It is not advisable todevelop a sample subset based on a pre-set percentage of data elements (e.g., 10percent of input data points). The riskof accepting a “bad” dataset can be unacceptably high for smaller data sets(e.g., N=100) when pre-set sample
percentages are used. To avoid this, it is recommended that sample sizes be set at 100 percent for N<200, and preset at
10 percent where N≥200.
Good Practice Recommendation:
When selecting sample subsets of data to perform the minimum QC procedures listed in Table 1 it isrecommended that sample sizes be set a 100 percent for N<200, and preset at 10 percent for N≥200.
Observed trends in errors should be noted as QC progresses and sample sizes should be increased
for larger data sets if necessary to correct the error.
Due to the quantity of data that needs to be checked for certain sources, automated checks are encouraged so as tosupplement manual checks and expand the sample size where possible. For example, one of the most common QC
activities involves verification that data keyed into a computer database are correct. A QC procedure could be set up to
use an automated range check (based on the range of expected values of the input data from the original reference) for
the input values as recorded in the database. Since this would verify only that the values in the database fall within therange of values in the original reference, the automated step can be supplemented with a representative sampling ofmanual checks on specific value transcriptions. Such combined manual and automated checks will typically be themost effective procedures in checking large quantities of input data.

510

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

TABLE 1RECOMMENDED MINIMUM REQUIRED QC PROCEDURES
QC Activity
Check the accuracy of data input from the originalreference source

Check that emissions are calculated accurately

Procedures
1.
2.

Confirm that correct references were used.
Cross-check sample of input data (either measurements orparameters used in calculations) for transcription errors.

1.

Use manual recalculation of a representative sample ofemission calculations.
Recalculation should cover complete set of calculations frombeginning to end.
Where necessary, mimic complex model calculations withabbreviated manual calculations to judge relative accuracy.

2.
3.

Check that parameter and emission units arecorrectly recorded and that appropriate conversion
factors are used

1.
2.
3.
4.
1.

Check the integrity of database files

2.
3.

Identify parameters (e.g., activity levels, constants) thatshould be common to multiple source categories and confirm
that there is consistency in the values used for theseparameters in emission calculations.

1.

Check that emissions data are correctly aggregated fromlower reporting levels to higher reporting levels when
preparing summaries.
Check that emissions data are correctly transcribed between
different intermediate products.

2.
1.

Internal Documentation Review

2.3

Confirm that path of data is correctly represented in database
and that all necessary processing steps are accounted for.
Confirm that data relationships are correctly represented indatabase.
Ensure that data fields are properly labeled and have correct
design specifications.

1.
Check for consistency in data sources where suchconsistency is expected

Check that the movement of inventory data amongprocessing steps is correct

Check that units are properly labeled in calculation sheets.
Check that units are correctly carried through from beginningto end of calculations.
Check that mass conversion factors are correct.
Check that temporal and spatial adjustment factors are usedcorrectly.

2.

Check that there is detailed internal documentation to support
the estimates and enable full reconstruction of the estimate.
Check that inventory data are archived and stored to facilitate
detailed review.

Source specific QC techniques

Source-specific QC techniques are those that are applied on a case-by-case basis, are directed at specific aspects of asource category, and require specific knowledge of the source. As opposed to general QC techniques, source-specific
QC techniques are directed at specific types of data used in the methodologies for individual sources. Due to thesedifferences, source-specific QC techniques require a fundamental understanding of the specific emission source andknowledge in the parameters associated with emissions from the source. From a technical skill level, thisdifferentiation between general QC techniques and source-specific QC techniques will therefore influence staffing
decisions.
The recommended source-specific QC activities include the following:•

Emission factor QC;•

Activity level QC, and•

QC of uncertainty estimates

The first two activities relate to the types of data used to prepare the emission estimate for a given source category. The
uncertainty estimate QC covers activities associated with determining uncertainties in emission estimates.

QA/QC of Inventory Systems

511

Background Paper

The actual QC that needs to be done by the country’s inventory agency will depend on the approach used to estimate
the emissions for a given source category. The first step in planning these activities is to determine where the inventory
estimates originated. If estimates are developed by outside agencies, the country’s inventory agency can, upon review,reference the QC activities of the outside agency as part of the QA/QC plan. There is no need to replicate QC activities
if the country’s inventory agency is satisfied that the QC activities performed by the outside agency meet the minimum
requirements recommended in this section.
The next step for implementing source-specific QC is to identify the type of data that are involved in the methodology
for the specific source being evaluated. After that has been determined, the following sections provide decision trees
for developing QC strategies directed at each of these types of data.

2.3.1

Emission factor QC

QC activities are organized around the three basic options for emission factor data: IPCC defaults, country-specificfactors, and factors based on direct emission measurements.
Figure 1 shows the decision tree for planning and implementing emission factor QC for IPCC defaults andcountry-specific factors. Figure 2 shows the decision tree for QC associated with methodologies that use directemission measurements from individual sites (either as the basis for a site-specific emission factor or directly for anemission estimate. The following sections describe the minimum recommended procedures for QC checks on IPCC
default factors, country-specific factors, and direct emission measurements.
IPCC default factors
Good Practice Recommendation:
The IPCC default factors have been reviewed prior to their publication in the IPCC Guidelines and
the QC associated with that review should be referenced directly.
As a supplemental QC activity, IPCC default emission factor checks could be expanded to include comparisons to siteor plant level factors within the country to determine the representativeness of the IPCC default factors to actual
sources in the country. This expanded check is strongly encouraged for countries that have site or plant level data
available for a source category, even if only for a small percentage of sites/plants. The check should be directed atsource categories that contribute a high percentage of emissions to the overall inventory and which also have a highlevel of uncertainty associated with them
Country-specific factors“Country-specific” refers to factors that are developed at a national or other aggregated level within the country basedon technology groups, local characteristics, or other criteria. They are not site-specific, but are used to represent anentire source category or subset of sources.
Where country-specific factors are used, two essential steps are necessary for emission factor QC. The first step is toevaluate the QC associated with secondary data used to develop the country-specific factor. Country-specific factors
are often not developed from primary testing by the inventory preparing agency. Instead, published studies andliterature sources (secondary data) are used to derive factors that can be used in the inventory. The common
characteristic of secondary data is that the data are not prepared and developed by the inventory preparing agency.
Therefore, it must be determined whether the QC activities conducted during the original preparation of the data areconsistent with the minimum required QC procedures outlined in Table 1 of this report.
Good Practice Recommendation:
The preparing agency should determine if the level of QC associated with secondary data at aminimum includes the same recommended QC procedures listed in Table 1. In addition, thepreparing agency should establish if any published data has received peer review of any type. The
presence of peer review would help establish the adequacy of the secondary data. If it is determinedthat the QC associated with the secondary data is inadequate, then the preparing agency must either
establish its own QC checks on the secondary data, or alternatively, reconsider the use of the data. If
it is determined that the QC associated with the secondary data is adequate, then the inventory
preparing agency can simply reference the data source for QC documentation.

512

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

Figure 1

Decision tree for emission factor QC

What is the Source of the
Emission Factor Data?

IPCC Default

Country-Specific
FactorCompare to site or 1
plant level factors todetermine
representativenessIs there sufficient QC
associated withsecondary data on
which country-specificfactor is based?

No

Establish
adequate QC
checks forsecondary data

Yes
Reference IPCC
guidelines for emission
factor data and keep
record of comparison

Yes

Does IPCC default
adequately represent
site or plant levelfactors?
No
Re-evaluate estimation
methodology todetermine if site/plantlevel factors can beincorporated

Does country-specificfactor compare
reasonably well with
IPCC default, or candifference be readily
explained?

No

Re-check factor
calculation for errors
and correct ifnecessary

Yes
Compare 1
country-specificfactor to site orplant levelfactors

Does country-specificfactor adequately
represent site or plantlevel factors?

Yes

Reference secondary
data QC activity andkeep record ofcomparisons to IPCC
default and site/plant
specific factors

No
Re-evaluate estimation
methodology todetermine if site/plantlevel factors can beincorporated
1 Comparison to plant/site level factors is a supplemental QC activity that is strongly encouraged where such data are

available.

QA/QC of Inventory Systems

513

Background Paper

Figure 2

Decision tree for QC of direct emission measurements

Are emissions measured
using internationallyrecognized standardized
methods?

Are the measurement
protocol considered
equivalent torecognized standard
methods?

No

Yes
Yes
Is there enough evidence toconfirm that measurement
protocol were correctly
implemented, and emissions
data are of sufficient quality

No

Yes

Does the site/plant have an
acceptable QA/QC plan foremissions measurements andare the results documented?
Yes
Do the results of QC activities
conducted at the site/plant
indicate emissions data are of
sufficient quality to be used?

No

Re-evaluate theuse of theemissions dataNoYes
Do emissions data (atsite/plant level and aggregatednationally) compare
reasonably well with historical
data or can differences bereadily explained?

Re-evaluate theuse of theemissions data forinventory

No

No

Investigate sites whereemissions estimates appear tobe outliers and review emission
data with site/plant personnel toidentify and correct errors

Yes
Do aggregated site-levelemissions compare reasonably
well with top-down estimatebased on IPCC default or candifferences be readily
explained?

No

Yes
Maintain site-level records ofstandard methods used andcomparisons to historical/topdown estimates. Reference
QA/QC plan and activities onrecord at individual sites

514

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

The second step in QC review of country-specific factors is to compare the country-specific factor to the IPCC default
to see if they compare reasonably well. The intent of this comparison is to see that the country-specific factor isconsidered reasonable relative to the IPCC default value. If the country-specific factor is significantly different fromthe IPCC default, this may point to an error in the factor calculation if the extremely large differences between thevalues cannot be explained otherwise. For example, for CO2 from fuel combustion, emission factors should be
expected to be fairly close in value. However, greater disparities might be explained for non-CO2 pollutants from fuelcombustion due to the uncertainty in the factors for these pollutants.
Good Practice Recommendation:
Compare the country-specific factor to the IPCC default for the source category. Record thedifference between the factors and maintain as documentation. If the difference is significant, thencheck the factor calculation for errors and correct if necessary.
A supplementary step for country-specific factor QC checks is to compare the country-specific factor to site or plantlevel factors. For example, if there are emission factors available for isolated plants (but not enough to support abottom-up approach) these plant-specific factors should be compared to the aggregated factor used in the inventory.
This type of comparison provides an indication of both the reasonableness of the country-specific factor and its
representativeness. This QC check is strongly encouraged for source categories that contribute a large percentage ofthe emissions in an inventory and which have a high level of uncertainty associated with the emission estimate.
Direct Emission Measurements
This QC review is directed at estimation methodologies that use direct emission measurements from individual
facilities (either as the basis for a site-specific emission factor or directly for an emission estimate). The checks
involved in this QC process establish whether measurements at the site were made according to recognized standards,
whether there is a QA/QC protocol in effect at the site, and whether the estimates compare reasonably well betweensites and top-down estimates.
Good Practice Recommendation:
The preparing agency should determine if emissions were measured using internationallyrecognized standards of measurement. If the measurements were not made using internationallyrecognized standards, determine if the measurement protocol used is equivalent to recognized
standard methods. If the measurement practices fail each of these criteria, then the emissions data
should be re-evaluated for use in the inventory.
Confirmation that internationally recognized, standard methods were used for measurements is highly recommended.
An example of the application of ISO standards can be seen in the area of emissions measurements and monitoring ofindustrial processes. ISO has published standards that specify procedures to quantify some of the performance
characteristics of air quality measurement methods such as bias, calibration, instability, lower detection limits,
sensitivity, and upper limits of measurement (ISO,1994). Such standards have direct application to QC activities
associated with emissions estimations from industrial sources based on measured values. If it is known that a facility
adheres to a set of known measurement standards for developing its emissions data, the inventory preparing agency
can use this knowledge in its assessment of data quality. In effect, the application of these standards by the industry
becomes part of the formal inventory QC process and can be referenced directly in the documentation of QC activity.
One example is where site-specific emission factors are used for a bottom-up, Tier 1 method for a large stationary fuelcombustion source. The inventory agency needs to establish the QC associated with the fuel measurements andanalysis that were made at the site to calculate the site-specific emission factor. If it is established that there isinsufficient QC control associated with the measurements and analysis used to derive the factor, continued use of thefactor may be questioned. Alternatively, if the plant has its own QA/QC plan, uses standardized measurement
methods, and its equipment is calibrated and maintained properly, the inventory agency should be more likely to
accept the site-specific factor for use in the calculation.
Supplemental QC activity is encouraged for bottom-up methodologies based on site-specific emission factors where
significant uncertainty remains in the estimates. Site-specific factors can be compared from site-to-site and also back to
IPCC or national level defaults. Significant discrepancies at particular sites may point to errors in the factor
calculations for those specific sites.

2.3.2

Activity level QC

Figure 3 shows the decision tree for activity level QC. The first step is to identify the source for the activity level data
used in the methodology. Activity data is normally developed at a national level (using secondary data sources) or

QA/QC of Inventory Systems

515

Background Paper

from site-specific data collected by the preparing agency. Site-specific activity data are usually prepared by the site orplant personnel from their own measurements. After the source of activity data has been determined, the remaining
steps can be followed.
National Level Activity Data
The estimation methodologies for many source categories rely on the use of activity data statistics that are not directly
prepared by the inventory agency. Where national activity data from secondary data sources are used in the inventory it
is critical for the inventory preparing agency to evaluate and reference QA/QC activities associated with the secondary
data preparation. This is particularly important in regards to activity data, since most activity data are originallyprepared for purposes other than input to air quality applications. Many statistical organizations, for example, havetheir own procedures for assessing the quality of the data independent of what the end use of the data may be. If it isdetermined these procedures satisfy the recommended minimum activities listed in Table 1, the inventory preparing
agency can simply reference the QC activity conducted by the statistical organization.
Good Practice Recommendation:
The preparing agency should determine if the level of QC associated with secondary data at aminimum includes the same recommend QC procedures listed in Table 1. In addition, the preparing
agency should establish if any published data have received peer review of any type. The presence ofpeer review would help establish the adequacy of the secondary data. If it is determined that the QC
associated with the secondary data is inadequate, then the preparing agency must either establish its
own QC checks on the secondary data, or alternatively, adjust its uncertainty estimates to reflect theresults of the QC review. If it is determined that the QC associated with the secondary data isadequate, then the inventory preparing agency can simply reference the data source for QC
documentation.
For example, in the transportation category, countries typically use either fuel usage or mileage statistics to develop
emission estimates. The national statistics on fuel usage and miles travelled by vehicles are usually prepared by aseparate agency of the government from that which prepares the inventory. However, it is the responsibility of theinventory preparing agency to determine what QA/QC activities were implemented by the agency who prepared theoriginal fuel usage and mileage statistics for vehicles. Questions to ask in this example are:•

Does the agency have a QA/QC plan that covers the preparation of the data?

•

What sampling protocol were used to estimate fuel usage/miles travelled?

•

Has any potential bias in the data been identified by the agency?

•

Has the agency identified uncertainties in the data?

•

Has the agency identified any errors in the data that may be carried over to the inventory?

The next QC check for national level activity data requires a comparison to the historical activity data for the source
being evaluated. Most source categories do not have dramatic changes in activity levels from one year to another, so ahistorical comparison should show a consistency in the prevalent trend of activity. If the estimated national activity
diverges greatly from this historical trend, either in sharp increases or decreases, the calculation of the current activity
should be checked for errors.
Good Practice Recommendation:
Compare current national level activity data to historical activity levels for previous five years. If
the current estimated activity level diverges greatly from the historical trend, evidenced either by a
sharp increase or decrease, the calculation of the current activity should be checked for errors andcorrected if necessary.
As a supplemental QC activity, a comparison check of activity data from multiple reference sources is encouraged.
This is strongly advised for sources that have a high level of uncertainty associated with their estimates. For example,
many of the agriculture sources rely on government statistics for activity data such as livestock populations, cultivatedacreage, and acres of prescribed burning. Similar statistics are often prepared by industry, university, or alternative
agencies which can be used to compare to the standard reference sources. These comparisons may need to be made ona regional level since many alternative references for such activity data will not cover the entire nation. However, theresults of such a comparison may point to inaccuracies in the standard national data set and prompt further review of
the activity level data.

516

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

Figure 3

Decision tee for activity level QC

What is the Source of Activity Data?

Establish
adequate
QC checksforsecondarydata

No

National Level
Activity from
Secondary Data
Sources

Site-Specific
Activity Data

Is there sufficient QC
associated with secondary
data source?

Does activity data compare
reasonably well from siteto-site or can differences
be readily explained?

Yes

Re-check
activity
data
calculationfor errors
and correct
ifnecessary

Does activity datacompare reasonably wellwith historical data, or candifference be readilyNo
explained?
Yes
Reference secondary data
QC activity and keep
record of comparisons tohistorical dataYes
Is there sufficient QC
associated with site-level
activity datameasurements?

No

Yes
Does aggregatedsite-level activity datacompare reasonably wellwith national levelstatistics (e.g., national
production, sales andimport/export data)?
Yes
Record results of site-tosite and national levelcomparisons; reference
QC associated with sitelevel measurements

QA/QC of Inventory Systems

No

Investigate sitesthat are outliersand correct
activity data
calculations if
necessaryNo

Consult withsite/plant
personnel toestablish
adequate QC
checks for
activity datameasurements

Investigate
cause fordifference
and
re-evaluate
methodology
if necessary

517

Background Paper

Site-specific Activity Data
Some methodologies rely on the use of site-specific activity level data used in conjunction with IPCC default or
country-specific emission factors. Site or plant personnel typically prepare these estimates of activity, usually for
purposes other than as inputs to air quality analysis. QC checks need to focus on inconsistencies between sites, either
due to errors or measurement techniques.
Good Practice Recommendation:
The inventory preparing agency should compare the activity data, adjusting for relative size orcapacity of the sites, on a site-to-site basis to identify significant outliers. If outliers are identified,they should be investigated to determine if the difference can be explained by unique characteristicsof the site or whether there is an error in the reported activity.
A variety of supplemental QC checks can be utilized to identify errors or misrepresentations of site level activity data.
Similar to the QC check on direct emission measurements, it is strongly encouraged that the inventory preparing
agency confirm that recognized standards were used in measurements for activity data at the individual sites. Site orplant personnel may have established QC procedures that can be directly referenced and included in the QC plan.
Comparisons of activity data from different reference sources can also be utilized to expand the activity level QC. For
example, in estimating PFC emissions from primary aluminium smelter, many countries use smelter-specific activity
data to prepare the inventory estimates. A QC check of the aggregated activity level from all aluminium smelters canbe made against national production statistics for the industry. Also, production data can be compared across different
sites, possibly with adjustments made for plant capacities, to evaluate the reasonableness of the production data.
Similar comparisons of activity data can be made for other manufacturing-based source categories where there ispublished data on national production.
Site-specific activity data checks can be applied to methodologies based on product usage. For example, good practicefor estimating SF6 emissions from use in electrical equipment relies on an account balancing of purchase of the gas,sales of gas for recycling, losses to handling, refills on maintenance, and total holding capacity of the system. Thisaccount balance system should occur at each facility where the equipment is in place. A QC check of overall national
activity could be made by comparing national SF6 production in the country (if such production exists) adjusted forimports and exports of SF6, to estimate total national usage of SF6. This total national usage can then be used as an
upper bound on SF6 emissions, since emissions from each of the sites relies on uncaptured losses of SF6. Similaraccounting techniques can be used as QC checks on other categories based on gas usage (such as the substitutes forozone-depleting substances) to establish upper boundaries on consumption and emissions.

2.3.3

QC of uncertainty estimates

The calculation of uncertainty estimates for each source category also must undergo QC. Good practices for
estimating uncertainties are being developed. These practices rely on calculations of uncertainty at the source level,
and then combined to summary levels for each sector and the entire inventory. Some of the methods rely on the use ofmeasured data associated with the emission factors or activity data to develop probability distribution functions fromwhich uncertainty estimates can be estimated. Many uncertainty estimates, however, will rely on expert judgement todevelop probability distribution functions and uncertainty ranges. In all cases, QC procedures should be applied to theuncertainty estimations to confirm that calculations are correct and that there is sufficient documentation to replicate
the development of the uncertainty estimate.
Good Practice Recommendation:
Confirm that uncertainty estimations have been documented for each source category. Cross-checkcalculations of uncertainty estimates to verify that there are no errors in the calculation ofsource-specific and combined estimates of uncertainty. For uncertainty estimates involving expert
judgement, confirm that there is documentation showing the data considered, literature references,assumptions made, scenarios considered, justifications for the probability distribution functions
obtained, and results from reviewers.

2.4

QA proceduresGood practice for QA procedures requires independent, objective review to assess the effectiveness of the internal QC
programme, the quality of the inventory, and to reduce or eliminate any inherent bias in the inventory process.

518

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

The use of expert peer review and independent audits is described below. Where there are formal stakeholder andpublic review mechanisms in place in a country, it is encouraged that these also be used to supplement expert peer
review and independent audits.
Expert Peer Review
Expert peer review consists of an independent review of calculations, assumptions, and/or documentation by aknowledgeable expert in the technical field. This is generally accomplished by reviewing documentation associatedwith the methodology and results, but usually does not include rigorous certification of data or references such as
might be done in an audit. The objective of the expert peer review is to ensure that assumptions and procedures used
are reasonable as judged by persons knowledgeable in a specific field.
Good Practice Recommendation:
Expert peer review should be applied to those source categories for which there is either a high levelof uncertainty in the emission estimate or the source is ranked high in importance due to emissions
contribution. The peer review should be performed by someone with a high level of expertise in theparticular source category. Expert peer review should be implemented at the following stages: (a)When estimation methodologies are first being adopted or revised; and (b) When the inventory iscompleted, but prior to its official or public release.
There are no standard tools or mechanisms for expert peer review, and its use should be considered on a case-by-casebasis. If there is a high level of uncertainty associated with an emission estimate for a source category, expert peer
review may provide information to improve the estimate, or at least, better quantify the uncertainty. If a source
category has a high emissions contribution to the overall inventory, regardless of the uncertainty associated with theestimate, the preparing agency should utilize expert peer review since even minor improvements to the estimate could
have a significant effect on the inventory.
When considering personnel for expert peer review it is important to remember that they do not necessarily have tohave experience in preparing emission inventories. The expertise should be related to the parameter that the inventory
agency wants the review focussed on. For example, the inventory agency may be entirely satisfied with its emission
factors for oil and natural gas production activities but has a lot uncertainty associated with its activity level estimates
for the category. Experts in the oil and gas industry may be identified that can review the activity estimates, regardlessof their experience with estimating emissions. Effective peer reviews often involve identifying and contacting key
industrial trade organizations associated with specific source categories. This expert input should be sought early in theinventory development process so that the experts can participate from the start. This is particularly true for source
categories for which new methodologies or revisions to methodologies are being considered.
The results of expert peer review, and the response of the inventory agency to those findings, can be extremely
important to widespread acceptance of the final inventory. All expert peer reviews should be well documented,
preferably in a report or checklist format that shows the findings and recommendations for improvement. Follow-up
peer review after corrections or revisions are made is essential to complete the peer review process.
Independent Audits
Audits are managerial tools used to evaluate how effectively the inventory preparation team complies with theminimum QC specifications outlined in the QC plan. An ‘independent’, sometimes referred to as ‘third-party’ or‘external’, audit is conducted by personnel who are not involved in the emissions inventory development process. It is
important that the auditor be independent of the inventory development team so as to be able to provide an objective
assessment of the processes and data evaluated.
Good Practice Recommendation:
Independent audits should be used whenever new emission estimation methodologies are adopted,
or when there are substantial changes to the existing methodology. Audits should be conducted todetermine that the inventory preparation team implemented all recommended minimum QC
procedures listed in Table 1 of this report. At a minimum, the audit should replicate the procedures
listed in Table 1
Audit activities replicate internal QC activities in order to verify that they were used. It is desirable for the auditor todevelop a schedule of audits at strategic points in the inventory development. For example, audits related to initial datacollection and measurement work should be conducted at points during or not long after these inventory steps arecompleted. In this way, deficiencies can be addressed before they affect the final product.

QA/QC of Inventory Systems

519

Background Paper

3

IMPLEMENTING THE QA/QC PLAN

There are key considerations that should be recognized before implementing any QA/QC activities. Decisions have to
be made as to which QA/QC techniques will be used, where they will be applied, and when they will be applied. There
are both technical and practical considerations in making these decisions.
Good Practice Recommendation:
The following considerations should be taken into account when implementing QA/QC activities:*

Priority of important sources: One of the key criteria in the good practice guidance for selecting
emissions estimation methods is the relative importance of the source category. The amount of
QA/QC activity devoted to a source category should reflect this criterion, and*Changes in methodology: Whenever new methods are adopted, or existing methods are revised,
there should be increased QA/QC activity associated with these changes. Such changes inmethodology may occur when making baseline recalculations and when splicing
methodologies.

Given the importance of focusing resources on the priority areas, it may be possible to minimize QA/QC activities insome areas. If there have been no substantial changes to the methodology for a source category, for example, it may bepossible to refer to previous QA/QC efforts regarding emission factor and activity level data sources. For example, if
fuel combustion estimates utilize the same standard national reference for fuel usage that was used in a previous
inventory submittal, and substantial QA/QC was performed on the previous inventory to ensure the quality of thenational reference fuel usage data, it may be satisfactory to reference the previous QA/QC effort for the activity data. In
such cases, other routine checks shown in Table 1 should still be performed on the emissions calculation.

4
4.1

REPORTING AND DOCUMENTATIONInternal documentation

Internal documentation refers to the full documentation of all QA/QC checks, audits, and reviews that are maintainedat the country level, but which can be produced if necessary for an on-site review. This includes any of the physicalrecords, such as the QA/QC plan, checklists, notes, calculation sheets, and reports that were utilized to conduct anddocument the QA/QC activity.
Good Practice Recommendation:
Records of QA/QC results should be retained for each source category where QA/QC activities havebeen implemented. There should be sufficient documentation of the results to indicate thechecks/audits/reviews that were performed, when they were performed, who performed them, andany corrections and modifications resulting from the QA/QC activity.

4.2

External documentation

External documentation refers to the documentation that is reported to outside parties as part of formal inventory
submittals or for end use of the data. The annex to the draft decision on UNFCCC Guidelines for the Preparation of
National Communications (FCCC/SBSTA/1999/L.5) specifically requests information on implemented QA/QC
procedures as part of national inventory reports for Annex I Parties. It is not practical to transfer all the internal
documentation that is retained at the country to outside agencies. Therefore it is necessary to summarize andcommunicate the key findings of the QA/QC programme.
A report summarizing the implemented QA/QC activities on a source category basis should be prepared. This report
should be submitted as part of the supplemental reporting material to the IPCC and as part of the national inventory
report to UNFCCC. There should be clear delineation in the report as to which activities were performed internally andwhich were done by external reviewers. Results should be summarized for each type of QA/QC activity performed.

520

Cross-cutting Issues

Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories

Good Practice Recommendation:
A summary report of QA/QC activity should be prepared. Following are the documentation items
that should be included in the summary QA/QC report:*

Descriptions of the specific QA/QC activities performed by the inventory agency for eachsource category;*

Document, by reference, any QA/QC activity not directly performed by the inventory agency but
which is utilized as part of the QA/QC process (e.g., the QA/QC programme of a national
statistics agency that provided activity data);*

Summary of QA/QC findings and resolution of problems by the inventory agency, and*Unresolved issues that affect inventory quality (e.g., confidentiality issues, data gaps,
unfinished QA/QC steps, etc.).

Much of the input to this QA/QC summary report can come directly from the written QA/QC plan, with additions
summarizing the results. The checklist provided in Table 1 of this report can be used to show what activities wereaccomplished. However, the report should only show those QA/QC activities that were actually implemented. It is notrecommended here that audit and general QA/QC checklists be submitted as part of this report due to the volume ofmaterial. Such checklists and detailed documentation of QA/QC activities should be maintained on record at thecountry as internal documentation, however.
In addition to the QA/QC summary report, it is recommended here that a notation key be added as a supplement to thequality assessment codes in the IPCC Overview Table for National Greenhouse Gas Inventories (Table 8A of the
IPCC Reporting Instructions). An additional field should be added to the table and labeled as “QA/QC”.

REFERENCES
Grant, E.L., and R.S. Leavenworth, 1988: Statistical Quality Control. McGraw-Hill Book Company, New York.
IPCC, 1997: Houghton, J.T., Meira Filho, L.G., Lim, B., Treanton, K., Mamaty, I., Bonduki, Y., Griggs, D.J.
and Callander, B.A. (eds), Revised 1996 IPCC Guidelines for National Greenhouse Gas Inventories (vols.
1-3), Paris.
IPCC, 1998: Expert Group Meeting on Managing Uncertainty in National Greenhouse Gas Inventories, Draft
Meeting Report. Meeting held in Paris, France, October 13-15, 1998.
IPCC, 1999: Expert Group Meeting on Good Practice in Inventory Preparation, General Background Paper.
Meeting held in Washington, D.C., United States, January 26-28, 1999.
ISO, 1994. Air Quality, Determination of Performance Characteristics of Measurement Methods. ISO
9196:1994. ISO, Geneva, 1994.
Lim, B., P. Boileau, Y. Bonduki, A.R. Van Amstel, L.H.J.M. Jansson, J.G.J. Oliver, and C. Kroeze, 1997.
Improving the Quality of National Greenhouse Gas Inventories. Summary report to the IPCC Expert Meetingon Methods for Assessment of Inventory Quality held in Bilthoven, The Netherlands, November 5-7, 1997.
UNFCCC, 1999. National Communications from Parties Included in Annex I to the Convention, Guidelines for
the Preparation of National Communications. FCCC/SBSTA/1999/L.5. 8 June 1999.
U.S. EPA, 1997. Emission Inventory Improvement Programme, Volume VI: Quality Assurance Procedures.
EPA-454/R-97-004F.

QA/QC of Inventory Systems

521

Background Paper

ANNEX IUSE OF ISO 9000: A DATA QUALITY MANAGEMENTSYSTEM FOR INVENTORIES

Data quality management systems establish standardized requirements and procedures to ensure that products meet therequirements of its end users. In the context of national GHG inventories, this includes a system of QC activities thatthe inventory preparer adopts to manage processes and factors that influence the quality of the emission estimates.
As part of its general QC procedures, it is encouraged that the Annex-1 Countries adopt the programme embodied in
the International Organization for Standardization (ISO) 9000 series, which provides standards for data documentation
and audits that could clearly be useful to the inventory QA/QC process. Some countries (e.g., the United Kingdom andNetherlands) have already adopted ISO 9000 standards either as part of their inventory development process or for the
management of their inventory data for end uses. The ISO 9000 series is not designed explicitly for emissions data
development and must be adapted for use to the inventory. Also, the adoption of ISO 9000 programme elements mustbe supplemented by source-specific QA/QC procedures.
Pertinent standards and guidelines published under the ISO 9000 series that should be considered for adoption ininventory development include:•

ISO 9004-1: General quality guidelines to implement a quality system;•

ISO 9004-4: Guidelines for implementing continuous quality improvement within your organization
using tools and techniques based on data collection and analysis;•

ISO 10005: Standard provides guidance on how to prepare quality plans for the control of specificprojects;•

ISO 10011-1: Guidelines for auditing a quality system;•

ISO 10011-2: Guidance on the qualification criteria for quality systems auditors;•

ISO 10011-3: Guidelines for managing quality system audit programmes;•

ISO 10012: Guidelines on calibration systems and statistical controls to ensure that measurements are
made with the intended accuracy, and•ISO 10013: Guidelines for developing quality manuals to meet specific needs.

At this time, it is encouraged that Annex-1 Countries utilize ISO 9000 series guidelines as self-assessment and quality
management tools and to clearly document this usage in their inventory submittal.

522

Cross-cutting Issues