Economic Statistics: Measurement Problems Can Affect the Budget and Economic Policy-Making
Released: 1995-05-02T00:00:00

Letter

Background

Economic statistics are intended to summarize various characteristics of the nation’s economy in an accurate and understandable way. Over the last decade, there has been growing concern among many experts that economic statistics have not kept pace with changes in the economy. Structural changes in the economy—such as new technologies, emerging industries, and demographic shifts—require statistical agencies to continually reassess old methodologies and data collection techniques and devise new ones. This process can be analytically complex and costly.

Responding to concerns about problems with federal economic statistics, President Bush, as a supplement to his first State of the Union message in February 1989, proposed an increase in federal spending to improve the quality and accuracy of business and economic statistics. As a result, a working group was established in April 1989 to assess problems within the federal economic statistical system and recommend needed improvements. The working group focused on recommending improvements that could be met in a timely manner with limited budget increases. The recommendations issued by the group were known as the Economics Statistics Initiative and were to be implemented in stages from fiscal years 1990 to 1996. Despite this initiative, many problems with economic statistics remain unresolved.

Objective, Scope, and Methodology

Our objective was to discuss how reported problems with selected statistics produced by the federal government can affect the federal budget and the formulation of the nation’s economic policy. To meet this objective, we first identified several federal statistical series that have direct effects on the federal budget and/or are used by government officials in the formulation and implementation of economic policy. Relying on our previous work, the work of experts on economic statistics, and interviews with officials from federal statistical agencies, we determined that many of these statistical series are subject to measurement problems to some degree. After consulting with your office, we decided to focus our work on selected statistical series that are used extensively in either the budget or policymaking process and which also have measurement problems that could be readily demonstrated. We chose (1) the CPI to illustrate how federal statistics can affect the federal budget and (2) international transaction and output and productivity statistics to show how federal statistics can affect the economic policymaking process. The problems of other statistics and their effects, however, may differ from those of the statistics we selected for our illustrations.

To develop these illustrations, we first discussed the problems with the selected statistics with officials from the agencies responsible for producing them. In addition, to obtain information on the agencies’ plans for addressing these problems, we reviewed the agencies’ budget and other submissions to OMB, prior studies and reports on economic statistics, and federal budget documents and analyses. We also judgmentally selected and reviewed several of the most recent academic studies and reports, congressional testimony, and other records documenting concerns with these statistics. We obtained views on the quality of the selected statistics from recognized experts outside of government.

Our second step in developing our illustrations was determining how identified problems with the selected statistics could affect the budget and economic policy. To do so, we first discussed potential effects of the identified problems with officials from the agencies that produce these statistics. We then examined budget and other documents to determine what effect, if any, problems with CPI could have on the budget. To analyze the sensitivity of federal spending and taxes to variations in consumer prices, we obtained assistance from officials at OMB and the Congressional Budget Office (CBO). To illustrate how problems with international transaction and output and productivity statistics affect economic policy, we obtained the views of agency officials and congressional staff involved in shaping policy on the economic activities measured by these statistics. We also reviewed our prior products that discussed CPI, international transaction statistics, and other topics related to this report. (They are listed at the end of this report.)

As agreed with your office, we did not attempt to identify the underlying causes or possible corrective actions for the problems with the selected federal statistics.

We obtained written comments on our report from the Department of Commerce where BEA and Census are located. We also obtained oral comments from the Office of Federal Statistical Policy within OMB, which is responsible for managing the federal statistical system. The Department of Labor, where BLS is located, was not able to meet our deadline for providing comments. However, BLS staff informally provided suggestions for technical clarifications to the report. We did our work primarily in Washington, D.C., from December 1993 to March 1995 in accordance with generally accepted government auditing standards.

The CPI Significantly Affects Federal Spending and Taxes

The CPI measures average price changes over time for consumer goods and services. BLS has reported that several measurement issues affect the accuracy of the CPI, and several recent studies and a number of prominent economists have said that the CPI overstates inflation. However, the studies do not provide firm support for a specific estimate of the overstatement or even a definitive demonstration that the CPI, in fact, overstates the true rate of inflation. The studies quantified the effect on CPI of some measurement problems but did not measure the effect of quality changes on prices, which according to BLS is the most serious CPI measurement issue. BLS officials said that without more information on the effect of quality on prices, a definitive estimate of the CPI’s misstatement of inflation is not possible.

Any misstatement of the CPI would have serious consequences. OMB and CBO estimated that even a small overstatement of the rate of inflation by the CPI could add billions of dollars to the federal deficit as a result of higher federal benefits payments and lower tax revenues. Some entitlement programs and income tax brackets are tied to the CPI to protect entitlement recipients and taxpayers from the effects of inflation.

The CPI Is Used as a Measure of Inflation

The CPI is a statistical measure of changes in prices of a fixed market basket of goods and services purchased by urban consumers. It is the only index compiled by the federal government that is designed to measure the changes in the purchasing power of the urban consumer’s dollar.

The CPI is used as a measure of the rate of inflation in the economy. The administration, Congress, and the Federal Reserve use trends in the CPI as an aid in formulating fiscal and monetary policies. Business and labor leaders as well as private citizens use the CPI as a guide to making economic decisions. The CPI is also used to adjust other measures for price changes, such as the income thresholds used by the government to determine the number of people living in poverty.

In addition, a version of the CPI is used to adjust some federal benefit payments for inflation. For example, in 1993, 42 million Social Security beneficiaries, 6 million Supplemental Security Income recipients, about 27 million food stamp recipients, and more than 3 million military and federal civilian retirees and survivors had their benefits adjusted for inflation, as a result of changes in the CPI. Finally, the CPI is used to adjust the federal individual income tax structure to prevent bracket creep, i.e., increases in real tax rates due solely to inflation. Benefit payments and tax brackets are adjusted automatically by the CPI, rather than on the basis of discretionary policy decisions.

Because the CPI is used to adjust benefit programs and tax brackets, it is often thought of as a cost-of-living index. However, according to BLS, while the CPI measures price changes, it is not intended to be a measure of changes in living costs. Like the CPI, a cost-of-living index would measure price changes between time periods. The two would differ, however, in what they would seek to measure. The CPI is designed to measure the price changes for a fixed market basket—the same amounts of the same goods and services—over time. A cost-of-living index, however, would allow changes in the mix of goods and services whose prices are measured. A cost-of-living index would thus recognize that prices for all goods and services do not change by a uniform amount, and that consumers are able to substitute one product for another as their relative prices change. An ideal cost of living index would measure how much more it would cost to maintain a constant standard of living or level of satisfaction even after such product substitutions. It could also measure how changes in areas such as environmental quality or public services would induce changes in consumer purchases of goods and services that would be included in the index. While such an index could be conceptually superior to the CPI as an inflation index for adjusting federal benefits and tax brackets, there are no generally accepted methodologies for calculating one.

BLS bases the CPI on a sample of prices of food and beverages, apparel, housing, transportation, medical services, entertainment, and other goods and services that consumers buy. These items are known collectively as the market basket. BLS conducts a continuous Consumer Expenditure Survey to determine which particular goods and services consumers are purchasing. This information is to subsequently be used when BLS revises the market basket. BLS generally does a revision every 10 years. BLS last did a revision in 1987, which reflected consumer spending patterns for 1982 through 1984.

BLS measures price changes each month by checking the prices of the sample of goods and services and then comparing the aggregate cost of the sample to the cost in the previous month. BLS obtains prices for most goods and services in the market basket through personal visits by its field representatives to approximately 21,000 retail establishments. BLS staff also sample about 60,000 housing units to obtain information on housing costs. In addition, BLS sponsors a survey of consumers, conducted by the Bureau of the Census, to determine which retail outlets its representatives should visit to obtain its monthly price quotations. This survey is conducted in approximately 20 percent of a sample of urban areas each year and, as a result, the entire sample is updated every 5 years.

BLS Researchers Have Reported That Measurement Issues Affect CPI Accuracy

In December 1993, BLS researchers reported that several measurement problems affect the accuracy of the CPI. According to BLS researchers, these measurement problems could cause the CPI to inaccurately gauge price changes for some products. One measurement problem is that the CPI may not adequately reflect current consumer spending patterns because BLS only does a revision of the market basket to reflect new consumer spending patterns about every 10 years. As a result, the current CPI may not fully reflect some current consumer spending patterns, such as the substitution of lower-priced goods for higher-priced ones.

Another measurement problem, according to BLS, stems from the procedures BLS uses to update the sample of retail outlets BLS field staff visit to obtain monthly prices for the items in the market basket. In recent testimony before the Committee on Finance, U.S. Senate, the BLS Commissioner noted that the procedures for systematically introducing new outlets and items into the CPI inadvertently tend to misstate the effect of temporary price increases. The procedures give higher weight than is justified to prices that are temporarily low in the month the new samples are introduced and lower weight than is justified to prices that are temporarily high. Thus, these procedures can cause an overstatement of price change in the period immediately following the sample update.

A third measurement problem is the difficulty in separating price changes resulting from quality changes from those due to inflation. Research indicated that price increases are sometimes the result of improvements to the quality of a product, such as upgrades to an automobile that make it safer or last longer. In these situations, consumers are paying more for a product, but they are getting a better product. BLS uses a number of methods to adjust the CPI to account for quality changes in products. However, BLS officials note that these methods may not adequately capture the effect of quality changes on the price of some products. Recent research suggests that BLS underadjusts the CPI for quality improvements to some products and overadjusts the index for quality improvements to other products. An underadjustment for quality would result in an overstatement of inflation while an overadjustment would result in an understatement. Some research also indicates that the CPI does not account for quality deterioration in some products and services, which would result in an understatement of inflation. BLS officials indicated that it is not known whether the overall measurement error associated with quality changes results in price increases being overstated or understated in the CPI. They said that more research is needed to improve the methodologies for separating the effect of quality modification price changes from inflation-related price changes.

Studies Indicate That the CPI Does Not Accurately Reflect Current Price Changes

BLS researchers and other price measurement experts disagree on the extent of the measurement problems surrounding the CPI. Several recent studies on the accuracy of the CPI indicated that it may overstate inflation. We reviewed four of the most recent of these studies. Each made a different estimate of the overstatement. A study by BLS researchers estimated that problems in measuring consumer spending patterns caused CPI to overstate inflation by 0.1 to 0.2 percent annually. A study by staff of the Federal Reserve Bank of Dallas concluded that the CPI overstated inflation by less than 1.0 percent annually, while CBO staff and Federal Reserve staff studies estimated annual overstatement ranges of 0.2 to 0.8 percent and 0.4 to 1.5 percent, respectively.

BLS officials emphasized that the authors of each of the four studies consider their estimates to be subject to great uncertainty. The estimates are based, in part, on projecting measurement problems that have been identified for specific components of the CPI to other components that may face similar problems. BLS officials noted that in making their estimate, the authors of the Federal Reserve Bank of Dallas study warned that “at present we simply do not know” the magnitude of CPI mismeasurement.

BLS Has Addressed Some Issues Affecting CPI Accuracy

Because of its concern for the accuracy of the CPI, over the years BLS has made changes to the CPI’s methodology. For example, in the 1980s, BLS changed its methodology for measuring changes in cost of shelter. It switched from measuring the costs associated with purchasing a house to measuring the cost of consuming the services provided by housing. Since the CPI is intended to measure consumption, this methodology allows BLS to remove the investment aspect of homeownership from the CPI and thus more accurately reflects the cost of shelter in the CPI. More recently, in January 1995, BLS incorporated new estimation techniques intended to further increase the accuracy of the CPI’s shelter, food-at-home, and prescription drug components.

BLS also has taken steps designed to improve its methodologies for measuring the effect of quality on price changes. Research had pointed to the possibility that quality-related problems may have resulted in the CPI understating inflation. For example, a 1988 study indicated that measurement problems associated with quality caused the CPI to understate inflation for several large CPI components including housing, new automobiles, and apparel. Since this study, BLS has changed its methodologies for measuring quality changes for the housing and apparel components of the CPI.

In addition, BLS has initiated an effort intended to improve the CPI’s ability to reflect changes in consumer spending patterns. In late 1994, BLS began a 6-year, $61 million project, part of which is to update the information on the percentage of consumption spending on particular items in the market basket. BLS officials told us that when the market basket update is finished, the CPI will be revised to better reflect current consumer spending patterns, which would address one of the measurement problems that tend to cause CPI to overstate inflation. The update is scheduled to be completed by January 1998.

Estimates of the Effects of CPI Overstatement on the Federal Budget

The CPI directly affects a significant portion of the budget because many federal benefits as well as individual income taxes are tied, or indexed, to the CPI in an effort to negate the effects of inflation. For example, during fiscal year 1994, 31 cents of every federal dollar spent, or $460 billion, and 44 cents of every dollar in tax revenue collected, or $550 billion, were subject to indexing to the CPI.

If the CPI overstated the rate of inflation, this overstatement would result in increased benefit payments and reductions in the amount of tax revenues the government collects from individuals’ taxes. Estimates by OMB and CBO indicated that even a small overstatement of inflation by the CPI may add billions of dollars to the federal deficit. For example, if the CPI overstated inflation by as little as 0.2 percentage point annually from 1995 through 1999, an estimated $19.1 billion would be added to the deficit over that 5-year period, according to OMB estimates and an estimated $21.8 billion, according to CBO estimates. If the CPI overstated inflation by 0.5 percentage point annually over the same 5-year period, the budget deficit would increase by about $47.8 billion, according to OMB estimates (See fig. 1.).

An understatement of inflation by the CPI would have the opposite effect of an overstatement on the budget.

Statistical Limitations Increase the Uncertainties That Face Policymakers

Economic statistics can also affect the policymaking process. While it is not possible to demonstrate that if the shortcomings of particular statistics were resolved, policymakers would have made different decisions, such shortcomings increase policymakers’ uncertainty about current economic conditions and long-term trends. In some cases, these shortcomings, which include limited coverage or outdated measures of some economic activities, preclude a full and up-to-date understanding of current economic conditions. Such shortcomings may also hinder analytical efforts to assess whether policies are working as intended. In the following sections, we discuss how outdated and incomplete measures of international trade and investment and economic output and productivity increase economic policymakers uncertainty.

Incomplete Coverage of Trade and Investment Flows Affected Policymakers’ Ability to Evaluate Trade Policy

International economic statistics are compiled primarily by BEA and Census. BEA compiles the balance of payments, a framework for estimating the flow of goods, services, capital, and other transfers between the United States and other countries. BEA uses data from Census, the Department of the Treasury, and other sources to produce the balance of payments. Census compiles statistics on merchandise trade, which are drawn from import and export documents collected by the U.S. Customs Service. These statistics are used in a variety of ways, ranging from guiding policymakers in trade negotiations and assessments of international macroeconomic conditions to the administration of trade agreements, trade programs, and trade damage determinations.

Statistics on international trade and investment have not fully kept pace with the changing patterns of international economic relations. As a consequence, the possible undercounting of U.S. exports has long been suspected. In addition, problems with import statistics have also been reported by us and others. While a number of steps have been taken in recent years to improve the accuracy of merchandise trade statistics,problems remain. For example, in 1993, as Congress considered the North American Free Trade Agreement, we testified that Census officials believed U.S. exports to Mexico were undercounted. Further, we subsequently reported that available data made it difficult to get a complete picture of U.S.-Mexico trade, including, for example, how much of U.S. imports from Mexico came from maquiladora manufacturing plants.

Statistics on services transactions lack adequate detail and coverage, according to statistical agency officials and users of trade statistics. While some improvements have been made in the collection of such statistics, the surveys that produce these statistics are still limited in frequency, level of detail, and coverage. Statistics are collected for about 30 different categories of services on the basis of surveys that vary in frequency, extent of coverage, and exemption levels. In comparison, merchandise trade statistics are based on a monthly count of roughly 14,000 separate categories of imports and nearly 8,000 export categories. We reported in 1994 that evaluating the economic impact of the services portion of the General Agreement on Tariffs and Trade was hampered by the lack of available and adequate data. The Industry Sector Advisory Committee on Services strongly recommended earlier in 1994 that U.S. services transactions statistics needed to be improved in order to monitor the agreement.

Officials at BEA and the Department of the Treasury readily acknowledged that similar to statistics on services transactions, measures of overseas private investment assets do not adequately reflect growth in this area. A 1992 National Research Council report concluded that the measurement of international investment flows was more likely prone to error than any other category of trade and investment. One factor contributing to this error was that BEA’s last benchmark survey of U.S. private investment in other countries, from which annual investment flows are calculated, was done in 1943. As a result, annual earnings accrued from overseas investment may have been underestimated by as much as $6 to $14 billion in 1988. Another factor contributing to measurement problems is that BEA does not track holdings of U.S. currency in other countries or new types of financial transactions, such as financial swaps, which have grown substantially.

Problems in the measurement of international trade and investment contribute to a sizable error, or discrepancy, in the U.S. overall balance of payments that limits the accuracy of information available to policymakers. In some years, the discrepancy was so large that it obscured the nature and direction of international flows. For example, in 1990, a large decrease in capital flowing to the United States, including a net outflow of $5 billion in private capital, contributed to concerns about the availability of capital in the United States. However, the size of the 1990 statistical discrepancy, an estimated $73 billion, made it difficult to determine whether the availability of capital was actually diminished.

Incomplete Output and Productivity Statistics Hinder Policymakers’ Ability to Understand Economic Conditions

One long-term goal of monetary and fiscal policy is to raise the living standards of the American people. In the long run, higher standards of living largely depend on rising economic output and productivity.However, according to statistical agency officials and many users of economic statistics, such as economists, key measures of economic output have not kept pace with changes in the economy. The primary problem they cited was the limited measurement of service sector output. This problem is particularly important because service sector output is estimated by BEA to have grown from 38 percent of the gross domestic product (GDP) in 1960 to more than half of the GDP today.

The growing importance of the service sector presents greater measurement and data collection challenges to statistical agencies. For example, steel production is easier to measure than the provision of services. Steel output is measured in tons produced, while the provision of services is measured less concretely, in terms of improved quality and convenience. A bank that extends lobby hours or provides automated teller services may not process more transactions. However, by expanding these services, the bank improves the quality and convenience of its banking services for its customers. In addition, certain service industries, such as the restaurant industry, have a large and changing universe of establishments compared to the small, well-known number of car manufacturers, thus making data collection more difficult.

Researchers also cited problems in accurately measuring productivity, particularly for the service sector. Zvi Griliches, a Harvard University professor and Director of Productivity Research for the National Bureau of Economic Research, has noted that sectors of the economy that are easier to measure, such as manufacturing and mining, have shown greater productivity gains than service sectors such as finance. However, he concluded that the difference may stem as much from difficulties in collecting appropriate statistics for measuring service sector productivity as it does from actual differences in productivity. For example, financial services output is an “imputed value” of labor input (i.e., the growth in output is not directly measured, and so the growth in labor inputs, such as hours worked, is used as a substitute measure). Accordingly, productivity measures for this sector should be interpreted with caution.

Users of output and productivity data said that more reliable data on output and productivity would be helpful in assessing policies to enhance future economic growth and higher living standards. For example, Federal Reserve Board staff said that more accurate measures of service sector output data would improve their ability to assess productivity gains and the reasons for them. Other users also noted that improved data would aid researchers in determining the effect of fiscal actions on output. This determination would provide a better basis for evaluating policy decisions in areas such as government sponsored research and development, training, and tax policy. For example, in commenting on the administration’s fiscal year 1995 budget, the House Committee on Science, Space and Technology noted that “it is extremely difficult to measure the economic and social benefits of (research and development) investments in dollar terms with any precision.” According to the Chief Economist for the Committee, “better data would help policymakers in making choices between different policy alternatives.”

Similarly, in 1993, we reported that an investment focus within the budget would provide a valuable supplement to the unified budget’s concentration on macroeconomic issues by directing attention to the consequences of choices within the budget for long-term economic growth. However, without accurate statistics on output and productivity, deciding among investment and consumption priorities within the budget becomes much more difficult.

Agency Comments

The Department of Commerce provided written comments on a draft of this report. Commerce said it had no substantive comments, but offered technical corrections, which we have made. Commerce recognized that measurement problems can affect economic policy-making, and said that the problems are being addressed by BEA and the Census Bureau. On March 30, 1995, we met with the Chief Statistician and an economist in OMB’s Office of Information and Regulatory Affairs, who generally agreed with the information presented in this report. They suggested some technical corrections and clarifications that we have incorporated. The Department of Labor offered oral comments from BLS. On several occasions, senior BLS officials suggested technical changes to the report concerning the CPI and productivity statistics that we have incorporated where appropriate. BLS officials noted that the BLS Commissioner’s March 13, 1995, testimony before the Senate Finance Committee (see p. 7 of this report) provides an extensive description of the measurement issues surrounding the CPI. They also said that the October 1992 Monthly Labor Review includes a discussion on the limited impact of service sector measurement procedures on the accuracy of the widely watched business sector productivity measures.

We are sending copies of this report to the Secretaries of Labor and Commerce, the Director of the Office of Management and Budget, and interested congressional committees. Copies also will be made available to others upon request.

If you have any questions regarding this report, please call me on (202) 512-8676. This report was prepared under the direction of James McDermott, Assistant Director, Federal Management Issues. Other major contributors to this report are listed in appendix I.

Major Contributors to This Report

General Government Division

Seattle Field Office

Office of the Chief Economist

Related GAO Products

The General Agreement on Tariffs and Trade: Uruguay Round Final Act Should Produce Overall U.S. Economic Gains (GAO/GGD-94-83b, July 1994).

Measuring U.S.-Canada Trade: Shifting Trade Winds May Threaten Recent Progress (GAO/GGD-94-4, Jan. 19, 1994).

U.S.-Mexico Trade: The Maquiladora Industry and U.S. Employment (GAO/GGD-93-129, July 20, 1993).

Budget Policy: Federal Capital Budgeting (GAO/T-AFMD-93-7, May 26, 1993).

U.S. Trade Data: Limitations of U.S. Statistics on Trade With Mexico (GAO/T-GGD-93-25, Apr. 28, 1993).

Customs Service: Trade Enforcement Activities Impaired by Management Problems (GAO/GGD-92-123, Sept. 24, 1992).

Federal Statistics: Merchandise Trade Statistics-Some Observations (GAO/OCE-89-1BR, Apr. 21, 1989).

Developing a Consumer Price Index for the Elderly (GAO/T-GGD-87-22, June 29, 1987).

Funds Needed To Develop CPI Quality Control System (GAO/GGD-83-32, Apr. 1, 1983).

A CPI for Retirees Is Not Needed Now but Could Be in the Future (GAO/GGD-82-41, June 1, 1982).

A Consumer Price Index for Retirees and Alternatives for Controlling Indexing (Testimony, Apr. 20, 1982).

Measurement of Homeowner Costs in the Consumer Price Index Should Be Changed (PAD-81-12, Apr. 16, 1981).

Alternatives for Modifying the Indexation of Federal Programs (Testimony, Mar. 10, 1981).

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Address Correction Requested