Climate Change: The Quality, Comparability, and Review of Emissions Inventories Vary Between Developed and Developing Nations
Released: 2010-08-05T00:00:00

Letter

Background

Greenhouse gases can affect the climate by trapping energy from the sun that would otherwise escape the earth’s atmosphere. Various human and natural activities emit greenhouse gases, with the production and burning of fossil fuels for energy contributing around two-thirds of man-made global emissions in 2005 (see fig. 1). The remaining third includes emissions from industrial processes, such as steel production and semiconductor manufacturing; agriculture, including emissions from the application of fertilizers and from ruminant farm animals; land use, such as deforestation and afforestation; and waste, such as methane emitted from landfills. Carbon dioxide is the most important of the greenhouse gases affected by human activity, accounting for about three-quarters of global emissions in 2005, the most recent year for which data were available.

Carbon dioxide (other)

The 14 nations in our study differ greatly in the quantity of their greenhouse gas emissions, the sources of those emissions, and their per- capita incomes. Emissions in 2005 ranged from about 7 billion metric tons of carbon dioxide equivalent in China and 6 billion metric tons in the United States, to about 300 million metric tons in Malaysia. The contribution of various sectors to national emissions also differed across nations, with emissions from energy and industrial processes accounting for more than 70 percent of emissions in most industrialized nations and 20 percent or less of emissions in Indonesia and Brazil (see fig. 2).

The Convention established a Secretariat that, among other things, supports negotiations, coordinates technical reviews of reports and inventories, and compiles greenhouse gas inventory data submitted by nations. The Secretariat has about 400 staff, located in Bonn, Germany, and its efforts related to national inventories are funded by contributions from the Parties. For the Secretariat’s core budget, Parties provided $52 million for the 2008-2009 budget cycle, of which the United States contributed $9.5 million ($3.76 million in 2008 and $5.75 million in 2009), excluding fees.

The Convention requires Parties to periodically report to the Secretariat on their emissions of greenhouse gases resulting from human activities. Parties do not generally measure their emissions, because doing so is not generally feasible or cost effective, and instead estimate their emissions. To help Parties develop estimates, the IPCC developed detailed guidelines—which have evolved over time—describing how to estimate emissions. The general approach is to use statistics on activities, known as activity data, and estimates of the rate of emissions per unit of activity, called emissions factors. For example, to estimate emissions from passenger cars, the inventory preparers could multiply the number of gallons of gasoline consumed by all cars by the estimated quantity of emissions per gallon. The IPCC guidelines allow nations to use various methods depending on their data and expertise. In some cases, with adequate data, estimates of emissions can be as accurate as direct measurements, for example for carbon dioxide emissions from the combustion of fossil fuels which contribute the largest portion of emissions for many nations.

The Parties agreed to the following five principles for inventories from Annex I nations: Transparent. Assumptions and methodologies should be clearly explained to facilitate replication and assessment of the inventory.

Consistent. All elements should be internally consistent with inventories of other years. Inventories are considered consistent if a Party uses the same methodologies and data sets across all years.

Comparable. Estimates should be comparable among Parties and use accepted methodologies and formats, including allocating emissions to the six economic sectors defined by IPCC—energy, industrial processes, solvent and other product use, agriculture, land-use change and forestry, and waste.

Complete. Inventories should cover all sources and sinks and all gases included in the guidelines.

Accurate. Estimates should not systematically over- or underestimate true emissions as far as can be judged and should reduce uncertainties as far as practical.

Annex I nations are to submit inventories annually consisting of two components—inventory data in a common reporting format and a national inventory report—both of which are publicly available on a Web site maintained by the Secretariat. The common reporting format calls for emissions estimates and the underlying activity data and emissions factors for each of six sectors—energy, industrial processes, solvent and other product use, agriculture, land-use change and forestry, and waste. It also calls for data on the major sources that contribute to emissions in each sector. The inventory data are to reflect a nation’s most recent reporting year as well as all previous years back to the base year, generally 1990. The 2010 reporting format called for nearly 150,000 items of inventory data and other information from 1990 through 2008. The common format and underlying detail facilitate comparisons across nations and make it easier to review the data by, for example, enabling automated checks to ensure emissions were properly calculated and to flag inconsistencies in data reported over time.

The national inventory report should explain the development of the estimates and data in the common reporting format and should enable reviewers to understand and evaluate the inventory. The report should include, among other things, descriptions of the methods used to calculate emissions estimates, the rationale for selecting the methods used, and information about the complexity of methods and the resulting precision of the estimates; information on quality assurance procedures used; discussion of any recalculations affecting previously submitted inventory data; and information on improvements planned for future inventories.

The Secretariat coordinates an inventory review process that, among other things, assesses the consistency of inventories from Annex I nations with reporting guidelines. The purposes of this process are to ensure that Parties are provided with (1) objective, consistent, transparent, thorough, and comprehensive assessments of the inventories; (2) adequate and reliable information on inventories from Annex I Parties; (3) assurance that inventories are consistent with IPCC reporting guidelines; and (4) assistance to improve the quality of inventories.

In supporting the inventory review process, the Secretariat provides scientific and technical guidance on inventory issues and coordinates implementation of Convention guidelines. Inventory reviews are supervised by the head of the reporting, data, and analysis program within the Secretariat. By June each year, the Secretariat checks each inventory for completeness and format, called an initial check, and conducts a preliminary assessment before submitting it to an inventory review team for examination. The Secretariat assembles inventory review teams composed of scientists and other experts from around the world to review inventories from all Annex I Parties according to the Convention’s review guidelines. The inventory review teams assess inventories in September by reviewing activity data, emissions factors, methodologies, and other elements of an inventory to determine if a nation has employed appropriate standards, methodologies, and assumptions to compute its emission estimates. From February through March, the inventory review teams develop inventory review reports outlining their findings.

In accordance with the Convention’s principle of common, but differentiated responsibilities, non-Annex I nations’ inventories’ format and frequency differ from those for Annex I nations. The reporting guidelines, which have evolved over time, encourage non-Annex I nations to use the IPCC methodological guidelines in developing their inventories, but do not specify that they must be used. While they submit inventories to the Secretariat, non-Annex I nations’ inventories are not stand-alone documents. Rather, a non-Annex I nation’s inventory is a component of its national communication, a report that discusses steps the nation is taking or plans to take to implement the Convention. Non-Annex I nations do not have to use the common reporting format or submit a national inventory report. Moreover, they do not submit an inventory each year, but instead the Parties to the Convention determine the frequency of their submissions. Parties have not agreed on a regular frequency for non- Annex I nations to submit their inventories.

Inventories From Seven Annex I Nations Were of Higher Comparability and Quality than Those From Seven Non-Annex I Nations Because of Several Barriers

According to expert inventory review teams, the 2009 greenhouse gas inventories of seven Annex I nations were generally comparable and of high quality, although some of their emissions estimates have substantial uncertainty. In contrast, we found that the most recent inventories from seven non-Annex I nations, although they met reporting guidelines, were of lower quality and generally not comparable. Finally, experts identified several barriers to improving inventory comparability and quality.

Recent Reviews Found That Selected Annex I Nations’ Inventories Were Comparable and of High Quality, but Some Estimates Have Substantial Uncertainty

All of the inventories submitted in 2009 by the seven selected Annex I nations were generally comparable and of high quality, according to the most recent inventory reviews conducted by expert review teams under the Convention. The reviews found that six of the seven nations— Australia, Canada, Japan, Russia, the United Kingdom, and the United States—used appropriate methodologies and data, employed reasonable assumptions, and did not systematically either over- or underestimate emissions in their 2009 inventories (covering data from 1990 through 2007). The one exception to this was Germany’s 2009 inventory, which the review team said did not follow guidelines for its agricultural emissions, in part because of its attempt to use newer methods. The change significantly reduced estimated emissions from agriculture, though the sector is a relatively small contributor to Germany’s total emissions. One inventory reviewer familiar with Germany’s 2009 inventory said its overall quality was fairly good. In addition, Germany appears to have addressed the issue of its agricultural emissions in its 2010 inventory submission by returning to its previous methods, which had the effect of increasing its estimates of emissions from agriculture. Experts said that the seven selected inventories were generally comparable, which means they generally used agreed-upon formats and methods. In addition, nine experts we interviewed said they were of high quality and did not have major flaws.

These findings show significant improvement in the seven nations’ inventories since our 2003 report. For example, we reported in 2003 that both Germany’s 2001 submission (covering data through 1999) and Japan’s 2000 submission (covering data through 1998) lacked a national inventory report, a critical element that explains the data and methods used to estimate emissions. Nearly all Annex I nations—including Germany and Japan—now routinely submit this report. In addition, the review team found Russia’s 2009 inventory showed major improvements. For example, Russia included a full uncertainty analysis for the first time and improved its quality assurance and quality control plan. Since our 2003 report, these 7 selected nations, and 34 other Annex I Parties, have submitted about seven inventories, which were generally on time and more comprehensive than previous inventories (see fig. 3).

The inventory review reports noted several potential problems that, while relatively minor, could affect the quality of emissions estimates. For example, the review of the 2009 U.S. inventory noted that assumptions about the carbon content of coal are outdated because they are based on data collected between 1973 and 1989. The effect on emissions estimates is not clear, but the carbon content of the coal burned as fuel may change over time, according to the inventory review report. Any such change would affect emissions, since coal is the fuel for about half of all U.S. electricity generation. The U.S. inventory also used a value from a 1996 agricultural waste management handbook to estimate nitrous oxide emitted from livestock manure. The inventory review noted that livestock productivity, especially for dairy cows, has increased greatly since 1996, which would also increase each animal’s output of nitrous oxide emissions. Using the IPCC’s methodology for calculating emissions from excreted nitrogen, we estimated that this would lead to an underestimate of roughly 4.7 percent of total nitrous oxide emissions and 0.2 percent of total greenhouse gas emissions. Finally, the review of Russia’s 2009 inventory noted that it did not include carbon dioxide emissions from organic forest soils, which the inventory review report said could be significant. The inventory reviews and one expert we interviewed attributed many of the potential underestimations to a lack of data or an adequate IPCC-approved methodology and said that nations were generally working to address the issues.

Even though the review teams found these seven inventories generally comparable and of high quality, the nations reported substantial uncertainty in many of the emissions estimates in their inventories. The term “uncertainty” denotes a description of the range of values that could be reasonably attributed to a quantity. All of the Annex I nations’ inventories we reviewed contained quantitative estimates of uncertainty. As shown in table 1, six of the seven nations reported uncertainties for their overall estimates between plus or minus 1 and 13 percent, and Russia reported overall uncertainty of about plus or minus 40 percent. That equates to an uncertainty of 800 million metric tons of carbon dioxide equivalent, slightly more than Canada’s total emissions in 2007. Russia’s relatively large uncertainty estimate could stem from several factors, such as less precise national statistics. In addition, Russia generally used aggregated national data rather than data that account for variation within the nation. This would increase uncertainty because aggregated data do not account for important differences that affect emissions, such as different types of technology used in the energy sector. Japan and Australia reported very low uncertainty in 2009. The inventory review report noted that Japan’s estimate was lower than estimates from other nations, but neither the report nor Japan’s inventory provides a full explanation. The review team for Australia said that its uncertainty ranges were generally consistent with typical uncertainty ranges reported for its sectors. Despite high levels of uncertainty in some instances, the inventory review teams found the seven inventories to be generally of high quality because the teams judge quality based on consistency with guidelines rather than strictly on the precision of the estimates.

The uncertainty of emissions estimates also varies among the different sectors of a nation’s economy. For example, uncertainty is relatively low for estimates of carbon dioxide emissions from the combustion of fossil fuels because the data on fuel use are generally accurate and the process that generates emissions is well understood. Uncertainty is much higher for certain categories within agriculture and land-use. For example, some nations report that the uncertainty in their estimates of nitrous oxide emissions from agricultural soils is greater than 100 percent, in some cases much greater. According to a March 2010 report by a National Research Council committee, this results from scientific uncertainty in emission factors. Table 2 shows the contribution of the most important sources of uncertainty in the U.S. inventory. The sources of uncertainty in the other six Annex I nations’ inventories follow a broadly similar pattern: the largest sources of uncertainty are either large sources of emissions—such as fossil fuel combustion and land use—or small but highly uncertain categories—such as agricultural soils.

Shortcomings in inventory reporting guidelines may decrease the quality and comparability of emissions estimates for land use, according to two experts we interviewed. For example, the guidelines state that nations should report all emissions from “managed forests,” but they have broad latitude in assigning forested land to this category. This choice may have a major effect on emissions; one expert said that it would be possible for some nations with large forested areas, such as Brazil, to offset all their emissions from deforestation by designating large areas of protected forest as managed and taking credit for all of the carbon dioxide absorbed by those forests. To address this potential inconsistency, the National Research Council committee report recommended taking inventory of all land-based emissions and sinks for all lands, not just man-made emissions on managed lands. Others said that designating land as managed forest remains the most practical way to estimate man-made emissions and removals because other methods are not well developed.

Inventories From Selected Non-Annex I Nations Met Guidelines but Were of Lower Quality and Generally Not Comparable

Inventories from the non-Annex I nations we reviewed met the Convention’s relevant reporting guidelines. All of the seven non-Annex I nations we reviewed—Brazil, China, India, Indonesia, Malaysia, Mexico, and South Korea—had submitted their first inventories. In addition, Mexico submitted its second, third, and fourth inventories, and South Korea submitted its second. Secretariat officials said the other selected nations could submit their second inventories, as part of their national communications, over the next few years. The reporting guidelines call for non-Annex I nations to estimate emissions for 1990 or 1994 in their first submission, and for 2000 in their second submissions, and to include estimates for carbon dioxide, methane, and nitrous oxide in all submissions. We found that all selected non-Annex I nations reported for relevant years and these three gases, but we did not assess whether nations used appropriate methodologies and assumptions to develop these estimates.

However, the seven inventories were generally not comparable and were of lower quality than inventories from Annex I nations in four ways: 1. Inventories from select non-Annex I nations were outdated. The most recent inventories from selected Annex I nations estimate emissions for 1990-2008. However, except for Mexico and South Korea, the most recently submitted inventories from selected non-Annex I nations are for emissions for 1994. (See figure 4.) 2. Some selected non-Annex I nations’ inventories do not estimate emissions of all gases. As shown in figure 4, inventories from China, India, Indonesia, and Malaysia did not include estimates of the emissions of synthetic gases. Independent estimates show that while synthetic gases were only 1 percent of global emissions in 2005, the emissions of synthetic gases increased by 125 percent between 1990 and 2005. Their emissions have also grown substantially in some non- Annex I nations, such as China, which had the largest absolute increase in synthetic gas emissions among all non-Annex I nations between 1990 and 2005, according to information from the International Energy Agency (IEA). 3. Select non-Annex I nations’ inventories, to varying degrees, lacked critical elements. We assessed inventories for several elements that, according to reporting guidelines, can improve the quality and transparency of inventories. First, only Brazil and Mexico provided a quantitative analysis of the uncertainty of their estimates. Second, we found that all inventories lacked adequate documentation of methodologies, emission factors, and assumptions and that most lacked descriptions of quality assurance and quality control measures. Third, none of the select nations reported in a comparable format, instead using different formats and levels of aggregation. For example, China estimated some methane emissions from various agricultural subsectors but grouped some of these estimates into only one category. In contrast, South Korea estimated these same emissions but reported them in separate categories. Overall, the lack of documentation and of a common reporting format limited our ability to identify and compare estimates across nations. Finally, only Mexico included an analysis of its key categories of emissions. 4. National statistics from some select non-Annex I nations are less reliable. According to three experts we interviewed and literature, some non-Annex I nations have less reliable national statistics systems than most Annex I nations. These systems are the basis for emissions estimates, and experts noted that the estimates are only as good as the underlying data. For example, researchers estimated that the uncertainty of carbon dioxide emissions from China’s energy sector was as high as 20 percent. In contrast, reported uncertainties in estimates of carbon dioxide emissions from fossil fuel use in many developed nations are less than 5 percent. In addition, the International Energy Agency noted a relatively large gap between its energy statistics and those used in the national inventories of some non-Annex I nations, highlighting a need for better collection of data and reporting of energy statistics by some non-Annex I nations.

Emissions from Fuel Combustion, 2009 Edition, I.5. between 1990 and 2005, which was about the annual emissions of Canada, Germany, Japan, and Russia in 2005 combined. Recognizing the importance of information from non-Annex I nations, in March 2010, a National Research Council committee recommended that Framework gorous inventory reporting and Convention Parties extend regular, rigorous inventory reporting and review to developing nations. review to developing nations.

National Research Council, Verifying Greenhouse Gas Emissions, 6.

Experts Identified Several Barriers to Improving the Comparability and Quality of Inventories

Experts we interviewed identified several barriers to improving the comparability and quality of inventories. First, 10 of the 12 experts who provided views about barriers said that a lack of data and scientific knowledge makes some types of emissions difficult to estimate for both Annex I and non-Annex I nations. For example, current estimates of emissions related to biological processes, such as those from agriculture and land use, can be uncertain because of limited data. Specifically, nations do not always collect data on livestock nutrition, which can affect methane emissions. In addition, emissions related to some biological processes are difficult to estimate because they are not fully understood or are inherently variable. Emissions related to agriculture, for example, depend on the local climate, topography, soil, and vegetation. In March 2010, a National Research Council committee recommended further scientific research and data collection to reduce the uncertainties in estimates of agriculture, forestry, and land-use emissions. Such emissions are important, contributing about one quarter of total global emissions in 2005, the most recent year for which global data were available. They are particularly important for some non-Annex I nations, where they can be the largest sources of emissions. In Brazil and Indonesia, for example, agriculture and land-use emissions accounted for about 80 percent of total emissions in 2005.

Second, 11 experts said that non-Annex I nations have limited incentives to produce better inventories. The current international system encourages Annex I nations with commitments under the Kyoto Protocol to improve their inventories. This is because their ability to participate in the Kyoto Protocol’s flexibility mechanisms—which provide a cost- effective way to reduce emissions—is linked to, among other things, the quality of certain aspects of their inventories. Late submissions, omissions of estimates, or other shortcomings can all affect nations’ eligibility to use these mechanisms. Therefore, low-quality inventories can affect nations’ ability to lower the costs of achieving their emissions targets. While four experts we interviewed said that this linkage between inventories and the flexibility mechanisms in the Kyoto Protocol has driven improvements in many Annex I nations’ inventories, incentives for non-Annex I nations are limited. Furthermore, four experts said that some non-Annex I nations may avoid additional international reporting because they see it as a first step toward adopting commitments to limit emissions.

In addition, experts and the national communications of selected non- Annex I nations identified several other barriers to improving the quality and comparability of inventories from non-Annex I nations, including: Less stringent reporting guidelines and lack of review. Reporting guidelines differ between Annex I and non-Annex I nations. Non-Annex I nations do not need to annually submit inventories or to report on as many gases, for as many years, with as much detail, or in the same format as Annex I nations. They also do not have to follow all IPCC methodological guidelines, although they are encouraged to do so. Six experts said that this less stringent reporting regime has contributed to the lack of quality and comparability in inventories from non-Annex I nations. In addition, non-Annex I nations have not benefited from the feedback of technical reviews of their inventories, according to one expert.

Financial and other resource constraints. Though eight experts generally said that many non-Annex I nations may lack needed financial and other resources, they differed on the magnitude and importance of additional international support. Non-Annex I nations may lack resources to improve data collection efforts, conduct additional research, or establish national inventory offices. The developed nations of Annex I provided the majority of about $80 million that has been approved for the latest set of national communications, which include inventories, from non-Annex I nations. However, one expert said that this has not been sufficient to fully support the activities needed. In their national communications, China and India indicated needing funding to, for example, improve data collection. Two experts said that improving non-Annex I nations’ inventories may require significant resources. On the other hand, others said that the funds involved may be relatively small, or that financial constraints may not be significant, at least for major non-Annex I nations. For example, according to a report from a National Research Council committee, significant improvements in inventories from 10 of the largest emitting developing nations could be achieved for about $11 million over 5 years. While experts disagreed about the importance of additional funding, three said that international funding should support capacity development in each nation. They said that more continuous support would improve on the current, project-based method of funding, which encourages nations to assemble ad-hoc teams that collect data, write a report, and then disband.

Lack of data and nation-specific estimates of emissions factors.

According to four experts and the Convention Secretariat’s summary of constraints identified by non-Annex I nations in their initial national communications, the lack or poor quality of data or a reliance on default emissions factors limit the quality of inventories. Most non-Annex I nations identified that missing or inadequate data was a major constraint for estimating emissions in at least one sector. For example, Indonesia reported that it did not estimate carbon emissions from soils because the data required were not available. Though inventory guidelines encourage the use of nation-specific emissions factors that reflect national circumstances, most non-Annex I nations use default values provided by the IPCC. The reliance on default values can increase uncertainties of estimates because national circumstances can differ significantly from the defaults. For example, Denmark’s nation-specific emission factor for methane emissions from sheep is twice as large as the default. Thus, if Denmark had used the default value, it would have underestimated its emissions from sheep by half.

The Inventory Review Process for Annex I Nations Has Several Strengths and Some Limitations, and No Comparable Process Exists for Non-Annex I Nations

Experts said that the process for reviewing inventories from Annex I nations has several notable strengths. They also identified three limitations, which may present challenges in the future. Moreover, we found that although the review process includes steps to help ensure the quality of reviews, there is no independent assessment of the process’ operations. Finally, there is no review process for inventories from non- Annex I nations.

The Process for Reviewing Inventories from Annex I Nations Has Several Strengths

Eight of the experts we interviewed said the process for reviewing inventories from Annex I nations has several notable strengths that enable it to generally meet its goals of providing accurate information on the quality of inventories and helping nations improve their inventories. (Figure 6 below depicts the inventory review process.)

Experts identified four broad categories of strengths: Rigorous review process. Five experts said the rigorous review process gives them confidence that review teams can identify major problems with inventory estimates. For example, the Secretariat and review teams compare data, emission factors, and estimates from each inventory (1) from year to year, (2) with comparable figures in other inventories, and (3) with data from alternative sources, such as the International Energy Agency (IEA) and the United Nations Food and Agriculture Organization. Reviewers also ensure methods used to estimate emissions are appropriate and meet accepted guidelines. In addition, IEA officials inform the inventory review process by reviewing energy data in inventories and independently identifying issues for review teams to consider further.

Qualified and respected reviewers. Three experts we interviewed said that well-qualified and widely respected inventory reviewers give the process credibility. Secretariat officials told us that a relatively small number of people in the world have the expertise to evaluate inventories without further training. Parties nominate reviewers, including leading scientists and analysts, many of whom are also inventory developers in their home nations. Reviewers must take training courses and pass examinations that ensure they understand inventory guidelines and appropriate methodologies before serving on a review team. Two experts said reviewers’ experience and qualifications allow them to assess the strengths and weaknesses in inventories, including whether nations use appropriate methodologies. This is particularly important because some nations use advanced or nation-specific approaches, which can be difficult to assess.

Capacity building. Three experts said the inventory review process builds expertise among reviewers from developed and developing nations. Specifically, they said the review process brings inventory specialists together from around the world, where they learn from each other and observe how various nations tackle challenges in compiling their inventories. Two experts said that reviewers return home and can use the knowledge and contacts gained from their review team experiences to improve their national inventories.

Constructive feedback. Two experts said that the inventory reviews provide constructive feedback to improve inventories from Annex I nations. This feedback includes identifying both major and minor shortcomings in inventories. Secretariat officials said that review teams, when they identify issues, must also offer recommendations for addressing them. For example, reviewers noted Russia’s 2009 use of default assumptions for much of its uncertainty analysis, and recommended that Russia develop values that better match the methods and data used in making the emissions estimates.

For these and other reasons, three experts we interviewed said that the review process has helped improve the quality of inventories from Annex I nations. Secretariat officials said that when review teams point out discrepancies or errors, many nations revise and resubmit estimates to correct problems. For example, Australia revised its estimates of carbon dioxide emissions from croplands after a review team pointed out that changes in croplands management affect emissions. Australia’s revisions decreased estimated emissions from croplands in 1990 by 138 percent, meaning the revisions had the effect of moving croplands from an estimated source of greenhouse gas emissions to a sink removing greenhouse gases from the atmosphere. For nations with Kyoto Protocol commitments, review teams may adjust estimates if they are not satisfied with a response to their findings. For example, the team reviewing Greece’s 2006 inventory concluded that estimates in several categories were based on methods, data, and emissions factors that did not adhere to reporting guidelines. The review team was not satisfied with Greece’s response, and recommended six adjustments to Greece’s estimates. These adjustments lowered Greece’s official baseline energy sector emissions by 5 percent, from 82 million to 78 million metric tons of carbon dioxide equivalent.

The Process for Reviewing Inventories from Annex I Nations Has Some Limitations

Experts, literature, and several nations identified some limitations of the review process, which may present challenges in the future if, for example, the process is expanded to incorporate non-Annex I nations. First, six experts we interviewed said the process does not independently verify emissions estimates or the quality of the underlying data. Review teams primarily ensure the consistency of inventories with accepted standards but do not check underlying activity data, such as the amount of fuel burned. Review teams do compare underlying data with those reported in other sources, but these other sources are not fully independent because they also come from the nations that supply the inventories. Two experts said that more thorough verification might involve comparing estimates to observed measurements or independently constructing estimates from raw data.

However, such approaches may be costly and, as a National Research Council committee reported, the other methods currently available do not allow independent verification of estimates. Furthermore, one expert said that the review of emissions estimates from agricultural soils and land-use sectors may be especially limited because of a lack of data and the inherent difficulty in measuring these emissions. The inability to more thoroughly assess inventories may reduce the reliability of review findings. For example, the inventory review process may have overlooked a significant shortcoming in at least one review. Specifically, in 2009, the national audit office of one Annex I nation found that its national inventory estimates may understate actual emissions by about a third because the inventory preparers used questionable statistics. The relevant agencies in that nation generally agreed with the audit office’s recommendations based on its assessment. The review for that inventory, however, did not identify this issue.

Second, four experts we interviewed and several nations have expressed concerns about inconsistency across reviews, though the magnitude of this potential problem is unclear. The concerns relate to the potential for review teams to inconsistently apply standards when assessing an inventory. Secretariat officials said the process of reviewing inventories involves some degree of subjectivity, since reviewers use professional judgment in applying inventory review guidelines to a specific inventory. As a result, review teams might interpret and apply the guidelines differently across nations or over time. Four experts we spoke with, as well as several nations, have raised such concerns. For example, the European Community reported that some nations have received, on occasion, contradictory recommendations from inventory review teams. Secretariat officials said lead reviewers are ultimately responsible for consistent reviews but that Secretariat staff assist the review teams during the process, and two Secretariat staff read through all draft inventory reports, in part to identify and resolve possible inconsistencies. In addition, lead reviewers develop guidance on consistency issues at annual meetings. The magnitude of this potential problem is unclear, in part because it has not been evaluated by an independent third party.

Third, three experts and officials we interviewed said there are not enough well-qualified reviewers to sustain the process. Three experts and Secretariat officials said that they did not know whether this shortage of available experts has affected the overall quality of reviews. The Secretariat has, in the past, reassigned staff and reviewers from work on national communications to the review of inventory reports, and it provides training to all reviewers to increase capacity and retain qualified reviewers. However, Secretariat officials said it may be difficult to sustain the quality of reviews in the future if the inventory review process is expanded to include inventories from non-Annex I nations without receiving additional resources, since this would substantially increase the demands on the review process.

The Review Process Does Not Have an Independent Assessment of Its Operations

The review process includes steps to help ensure the quality of reviews, but we found that its quality assurance framework does not independently assess the process. Secretariat officials said that lead reviewers oversee the drafting of review reports, and review officers, lead reviewers, and review teams maintain a review transcript to keep track of potential issues they have identified with inventories, of nations’ responses to those issues, and of their resolution. However, lead reviewers, in the report of their 2009 meeting, expressed concern that these review transcripts are sometimes incomplete and are not always submitted to the Secretariat. In providing information on their experience with the review process and recommendations for improvements, the nations of the European Community suggested in late 2008 that the review process would benefit from establishing clear quality assurance and quality control procedures as well as from an annual analysis of its performance in relation to its objectives. Secretariat officials said they designated a Quality Control Officer who, along with the supervisor of the review process, reads all draft review reports and may identify problems and check underlying information in reports. Furthermore, Secretariat officials said that lead reviewers meet annually to discuss the review process, assess and prepare guidance about specific issues or concerns about the review process, and develop summary papers to report to Parties.

Nonetheless, the review process lacks an independent assessment of its operation. We examined several other review processes and found that periodic external assessments by independent entities can provide useful feedback to management and greater assurance that the review processes are working as intended. Inventory guidelines call for Annex I nations to carry out quality assurance activities for their own inventories, including a planned system of reviews by personnel not directly involved in the process. Though some United Nations and Framework Convention oversight bodies have the ability to assess the inventory review process, none have done so. The Secretariat has internal auditors, but they have not audited the inventory review process and Secretariat officials said they did not know of any plans to do so. Although the Compliance Committee of the Kyoto Protocol has reviewed aspects of the review process, issuing a report with information on consistency issues, this report was not a systematic review and was not developed by people independent of the review process.

No Inventory Review Process Exists for Non- Annex I Nations

As stated earlier, inventories from non-Annex I nations do not undergo formal reviews. The Secretariat compiled a set of reports summarizing inventory information reported by non-Annex I nations, such as inventory estimates, national circumstances, and measures to address climate change. However, Secretariat officials said they had not assessed the consistency of non-Annex I nations’ inventories with accepted guidelines. These officials also said that they did not plan to compile another report covering non-Annex I nations’ second inventories because the Parties have not agreed to this. An expert we interviewed said that the quality of inventories from non-Annex I nations is unknown because their inventories have not been formally reviewed. Two experts said that some non-Annex I nations have resisted increased scrutiny of their inventories because of sovereignty concerns, meaning that nations do not want to disclose potentially sensitive information or data to other political bodies. The growth in greenhouse gas emissions along with lower quality inventories in some non-Annex I nations is likely to increase the pressure for a public review of their inventories in the future.

Experts Said the Inventory System Is Generally Sufficient for Monitoring Compliance with Current Agreements, but Future Agreements with Non- Annex I Nations Could Pose Challenges

Most experts we interviewed said that the inventory system for Annex I and non-Annex I nations is generally sufficient for monitoring compliance with current agreements. However, they said that the system may not be sufficient for monitoring non-Annex I nations’ compliance with future agreements that include commitments for them to reduce emissions.

Experts Said the Inventory System Supports Current Agreements and International Negotiations

Eleven of the experts we interviewed said the inventory system— inventories and the process for reviewing them—is generally sufficient for monitoring compliance with current agreements, though five raised some concerns. All 11 of the experts who provided their views on the implications of the inventory system expressed confidence that inventories and the Convention’s inventory review process are suitable for monitoring Annex I nations’ compliance with existing commitments to limit emissions. In part, this is because emissions in many Annex I nations primarily relate to energy and industrial activity, which can be more straightforward to estimate and monitor than emissions from land use and agriculture.

Nevertheless, five experts raised at least one of two potential challenges facing the current system. First, three said they were cautious until they see how the system performs under the more demanding conditions of submitting and reviewing inventories that will show whether nations have met their binding emission targets under the Kyoto Protocol. When inventories are for years included in the Protocol’s commitment period, nations may be more concerned about meeting emissions targets, and review teams may face pressure to avoid negative findings. Second, three experts said that flexibilities in the current inventory system or difficulties in measuring and verifying emissions from some agriculture and land-use segments could create complications for international emissions trading under the Kyoto Protocol. Emissions trading under the Kyoto Protocol allows nations with emissions lower than their Kyoto targets to sell excess allowances to nations with emissions exceeding their targets. Though Parties to the Kyoto Protocol developed and agreed to the current system, three experts indicated that ensuring greater comparability of estimates between nations and types of emissions might be useful for emissions trading. For non-Annex I nations, eight experts said that their lower quality inventories and lack of review do not present a current problem since these nations do not have international commitments to limit their emissions.

Seven of the experts said that the inventory system is sufficient to support international negotiations. To develop agreements, two experts said, negotiators need information on current and historic emissions from the nations involved. Annex I nations submit this information in their annual emissions inventories, the most recent of which cover emissions from 1990 to 2008. Although emissions estimates in most non-Annex I nations’ inventories are outdated, seven experts said that there are enough independent estimates to provide negotiators with adequate information. State officials said that independent estimates are useful, but official national inventories would be preferable because they can lead to more constructive discussions and can help create capacity in nations to better measure emissions. In international negotiations, State has emphasized the need for better information on emissions from all high-emitting nations, including non-Annex I nations.

Experts Said the Inventory System May Not Be Sufficient for Monitoring Non-Annex I Nations’ Compliance with Future Agreements, Though Other Barriers Pose Greater Challenges

Different types of commitments would place different demands on the inventory system. Thus, the implications of the state of the inventory system for a future agreement will largely depend on the nature of that agreement. For Annex I nations, eight experts said that future commitments were likely to resemble current commitments and therefore the inventory system is likely to be sufficient. However, for non-Annex I nations, if future agreements include commitments to limit emissions, the current system is not sufficient for monitoring their compliance, according to nine experts. This is because non-Annex I nations do not submit inventories frequently, the quality of their inventories varies, and they do not undergo an independent technical review. Additional reporting and review could pose challenges since it could take time for non-Annex I nations to improve their inventories and Secretariat officials said that adding non-Annex I nations to the current inventory review process could strain the capacity of that system.

Some types of commitments by non-Annex I nations could be especially difficult to monitor and verify, according to experts. In the nonbinding 2009 Copenhagen Accord, many nations submitted the actions they intended to take to limit their greenhouse gas emissions, with Annex I nations committing to emissions targets for 2020 and non-Annex I nations announcing various actions to reduce emissions. Experts identified several challenges with monitoring the implementation of some of the actions proposed by non-Annex I nations (see table 3). For example, two experts said that monitoring emissions reductions from estimates of future business-as-usual emissions may prove challenging. They said this is because such actions may require Parties to estimate reductions from a highly uncertain projection of emissions that would have otherwise occurred. Parties would also have to develop and agree on guidelines to estimate and review business-as-usual emissions in addition to actual emissions. Similarly, monitoring reductions in the intensity of greenhouse gas emissions—emissions per unit of economic output, or gross domestic product—could pose challenges because of uncertainties in estimates of gross domestic product. One expert said that these challenges arise because the Parties to the Convention created the current inventory system to monitor compliance and evaluate progress among Annex I nations with national targets. This expert added that Parties to a new agreement may need to supplement the system to support the types of actions under consideration by non-Annex I nations.

Eight of the experts we interviewed said that Parties to a future agreement could overcome or mitigate many of the challenges related to inventories. For example, two experts said that Parties could design agreements that rely less on emissions estimates that are inherently uncertain or difficult to verify. For example, quantitative targets could apply only to sectors or gases that are relatively easy to measure and verify, such as carbon dioxide emissions from the burning of fossil fuels.

Three experts said that barriers other than the inventory system pose greater challenges to designing and reaching agreements on climate change. For example, nations disagree on the appropriate emissions limits for developed and developing nations. According to three experts, such disagreements were more of an obstacle to a comprehensive agreement in the latest round of negotiations in Copenhagen than were inventory issues. In addition, one expert pointed out that Parties to international agreements generally have limited ability to get other Parties to comply. For example, at least one nation with a binding emissions target under the Kyoto Protocol is unlikely to meet its target based on current inventory estimates and policies, according to this expert. Nations may be reluctant to agree to an international agreement until they have some assurance that other nations will follow through on their commitments.

Conclusions

High quality and comparable information on national greenhouse gas emissions is critical to designing and implementing international responses to climate change. The nations we reviewed meet their inventory reporting obligations, and review reports indicate this has resulted in generally high quality inventories from the seven highest emitting Annex I nations. However, the current inventory system does not request high quality emissions information from non-Annex I nations, which account for the largest and fastest growing share of global emissions. We found that the inventories from seven selected high emitting non-Annex I nations were generally outdated, not comparable, and of lower quality than inventories from Annex I nations. The existing gap in quality and comparability of inventories across developed and developing nations makes it more difficult to establish and monitor international agreements, since actions by both developed and developing nations will be necessary to address climate change under future international agreements. As a recent National Research Council committee study pointed out, extending regular reporting and review to more nations may require external funding and training, but the resources needed for the largest emitting developing nations to produce better inventories is relatively modest.

While our work suggests that the current inventory review process has notable strengths, we identified limitations that may present challenges in the future. For example, some experts and nations have reported concerns about inconsistent reviews and that resources may not be sufficient in the future. Stresses on the review process are likely to increase as review teams begin to review inventories that cover years in which some nations have binding emissions targets and if inventories from non-Annex I nations are subjected to inventory review under a future agreement. The Convention Secretariat has internal processes in place to help ensure quality reviews, but no systematic independent review to assess the merits of concerns about the consistency of reviews or to assess the need for additional qualified reviewers in the future. Addressing these issues could benefit the Secretariat by further enhancing confidence in its processes and ensuring that it has the resources necessary to maintain high quality reviews.

Recommendations for Executive Action

We are making two recommendations to the Secretary of State: 1. Recognizing the importance of high quality and comparable data on emissions from Annex I and non-Annex I Parties to the Convention in developing and monitoring international climate change agreements, we recommend that the Secretary of State continue to work with other Parties to the Convention in international negotiations to encourage non-Annex I Parties, especially high-emitting nations, to enhance their inventories, including by reporting in a more timely, comprehensive, and comparable manner, and possibly establishing a process for reviewing their inventories. 2. To provide greater assurance that the review process has an adequate supply of reviewers and provides consistent reviews, we recommend that the Secretary of State, as the U.S. representative to the Framework Convention, work with other Parties to the Convention to explore strengthening the quality assurance framework for the inventory review process. A stronger framework could include, for example, having an independent reviewer periodically assess the consistency of inventory reviews and whether the Secretariat has sufficient resources and inventory reviewers to maintain its ability to perform high quality inventory reviews.

Agency Comments and Third-Party Views

We provided State, the Convention Secretariat, and EPA with a draft of this report for review and comment. State agreed with our findings and recommendations and said that the department has been working with international partners in negotiations and through bilateral and multilateral partnerships to support and promote improved inventory reporting and review. State’s comments are reproduced in appendix III.

The Convention Secretariat provided informal comments and said that it appreciated our findings and conclusions. The Secretariat said that the report provided a comprehensive overview of the existing system for reporting and reviewing inventories under the Convention and the Kyoto Protocol, as well as very useful recommendations on how this system could evolve in the future and steps to be taken to that end. The Secretariat noted our acknowledgement of the strengths of the inventory review process for Annex I nations. In addition, the Secretariat commented on our discussion of the limited availability of statistics against which to compare inventory data, saying that this lack of data does not imply that its review process lacks independent verification and that its review teams rely on available statistics in conducting their reviews. The Secretariat also said that the disparities in inventory quality across Annex I and non-Annex I nations should be viewed in the context of the “common but differentiated responsibilities” of developed and developing nations under the Convention.

In addition, EPA and the Convention Secretariat provided technical comments and clarifications, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, Secretary of State, Administrator of EPA, Executive Secretary of the Convention Secretariat, and other interested parties. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or stephensonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Our review provides information on: (1) the comparability, quality, and barriers to improving inventories submitted by developed and developing nations to the United Nations Framework Convention on Climate Change (the Convention); (2) the strengths and limitations of the Convention’s inventory review process; and (3) the views of experts on the implications for agreements to reduce greenhouse gas emissions. To address all of these objectives, we reviewed relevant literature and Convention documents; met with officials from the Environmental Protection Agency (EPA), Department of State (State), the Convention Secretariat, and others to understand inventories, the inventory review process, and international negotiations; and summarized the views of experts on these issues.

Specifically, to address the first objective, we selected a nonprobability sample of 14 nations, seven Annex I nations—Australia, Canada, Germany, Japan, Russia, the United Kingdom, and the United States—and seven non- Annex I nations—Brazil, China, India, Indonesia, Malaysia, Mexico, and South Korea—based on the size of their emissions (including emissions from land-use and land-use change and forestry). We selected the largest emitting Annex I nations. For non-Annex I nations, we selected the largest emitting nations who had submitted inventories based on data available at the time. We omitted Myanmar because it did not submit an inventory to the Convention. We also ensured coverage of major variations in selected nation’s income and sectoral structure of their economies. To illustrate this variation, we used the World Bank’s data on per capita income levels, and data from the World Resources Institute and Convention Secretariat on emissions from the energy and industrial processes sectors. The selected 14 nations represented about two thirds of the world’s greenhouse gas emissions not related to land use and forestry in 2005. Our findings are not generalizable to other nations because the selected nations are not necessarily representative.

To assess the comparability and quality of inventories from Annex I nations, we summarized the results of the Convention’s 2009 reviews of inventories from selected Annex I nations, the most recent reviews available. We did not independently assess the validity of data, assumptions, or methodologies underlying the inventories we reviewed. Though we identified some limitations with the inventory review process, we believe that reviews provide reasonable assessments of the comparability and quality of inventories from selected Annex I nations. For non-Annex I nations, we assessed whether the latest inventories from selected nations included estimates for all major greenhouse gases (carbon dioxide, methane, nitrous oxide, hydroflurocarbons, sulfur hexafluoride, and perfluorocarbons), for all sectors (energy, industrial processes, solvent and other product use, agriculture, land-use change and forestry, and waste) and various years, and checked for inclusion of key inventory characteristics, including descriptions of uncertainty and quality assurance and quality control measures, adequate documentation to support estimates, a comparable format, and analysis to identify emissions from key sources. Though inventory guidelines do not call for all of these from non-Annex I nations, we believe they are indicative of the quality and comparability of inventories. We did not independently assess emissions estimates from non-Annex I nations. We used the quality principles agreed to by Parties for Annex I nations—transparency, consistency, comparability, completeness, and accuracy—as the basis of our review of all inventories and in our discussions with experts. We also provide information on the reported uncertainty of emissions estimates, a more objective indicator of their precision, and on the timeliness of inventory submissions. To identify barriers to improving inventories, we reviewed relevant literature, including national communications from the seven selected non-Annex I nations, and summarized the views of our expert group.

To address the second objective, we summarized the results of semi- structured interviews with experts and Secretariat officials. We reviewed Convention documentation about the inventory review process, including Compliance Committee and Subsidiary Body for Implementation reports.

To address all three objectives, we summarized findings in the literature and the results of semi-structured interviews with experts. First, we identified 285 experts from our review of the literature and recommendations from U.S. and international government officials and researchers. From this list, we selected 15 experts based on (1) the relevance and extent of their publications, (2) recommendations from others in the inventory field, and (3) the extent to which experts served in the Consultative Group of Experts (a group assembled by the Convention to assist non-Annex I nations improve their national communications), as lead reviewers in the Convention’s inventory review process, or were members of the National Research Council’s committee on verifying greenhouse gas emissions. Finally, to ensure coverage and range of perspectives, we selected experts who had information about key sectors, like the agriculture and energy sectors, came from both Annex I and non- Annex I nations and key institutions, and provided perspectives from both those who were involved in the inventory review process and from those not directly involved in preparing or reviewing inventories. Appendix II lists the experts we interviewed, which included agency and international officials, researchers, and members of inventory review teams. We conducted a content analysis to assess experts’ responses and grouped responses into overall themes. The views expressed by experts do not necessarily represent the views of GAO. Not all of the experts provided their views on all issues. We identify the number of experts providing views where relevant.

During the course of our review, we interviewed officials, researchers, and members of inventory review teams from State, EPA, and the Department of Energy in Washington, D.C.; the Convention Secretariat’s office in Bonn, Germany; and from various think tanks, nongovernmental organizations, and international organizations.

We conducted this performance audit from September 2009 to July 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: List of Experts

Appendix III: Comments from the Department of State

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, Michael Hix (Assistant Director), Russell Burnett, Colleen Candrl, Kendall Childers, Quindi Franco, Cindy Gilbert, Jack Hufnagle, Michael Kendix, Thomas Melito, Kim Raheb, Ben Shouse, Jeanette Soares, Kiki Theodoropoulos, Rajneesh Verma, and Loren Yager made key contributions to this report.

Related GAO Products

Climate Change: Observations on Options for Selling Emissions Allowances in a Cap-and-Trade Program. GAO-10-377. Washington, D.C.: February 24, 2010.

Climate Change Policy: Preliminary Observations on Options for Distributing Emissions Allowances and Revenue under a Cap-and- Trade Program. GAO-09-950T. Washington, D.C.: August 4, 2009.

Climate Change Trade Measures: Estimating Industry Effects. GAO-09-875T. Washington, D.C.: July 8, 2009.

Climate Change Trade Measures: Considerations for U.S. Policy Makers. GAO-09-724R. Washington, D.C.: July 8, 2009.

Climate Change: Observations on the Potential Role of Carbon Offsets in Climate Change Legislation. GAO-09-456T. Washington, D.C.: March 5, 2009.

Climate Change Science: High Quality Greenhouse Gas Emissions Data are a Cornerstone of Programs to Address Climate Change. GAO-09-423T. Washington, D.C.: February 24, 2009.

International Climate Change Programs: Lessons Learned from the European Union’s Emissions Trading Scheme and the Kyoto Protocol’s Clean Development Mechanism. GAO-09-151. Washington, D.C.: November 18, 2008.

Carbon Offsets: The U.S. Voluntary Market is Growing, but Quality Assurance Poses Challenges for Market Participants. GAO-08-1048. Washington, D.C.: August 29, 2008.

Climate Change: Expert Opinion on the Economics of Policy Options to Address Climate Change. GAO-08-605. Washington, D.C.: May 9, 2008.

International Energy: International Forums Contribute to Energy Cooperation within Constraints. GAO-07-170. Washington, D.C.: December 19, 2006.

Climate Change: Selected Nations’ Reports on Greenhouse Gas Emissions Varied in Their Adherence to Standards. GAO-04-98. Washington, D.C.: December 23, 2003.

Climate Change: Information on Three Air Pollutants’ Climate Effects and Emissions Trends. GAO-03-25. Washington, D.C.: April 28, 2003.

International Environment: Expert’s Observations on Enhancing Compliance With a Climate Change Agreement. GAO/RCED-99-248. Washington, D.C.: August 23, 1999.

International Environment: Literature on the Effectiveness of International Environmental Agreements. GAO/RCED-99-148. Washington, D.C.: May, 1999.

Global Warming: Difficulties Assessing Countries’ Progress Stabilizing Emissions of Greenhouse Gases. GAO/RCED-96-188. Washington, D.C.: September 4, 1996.