Retirement Security: BLS Should Explore Ways to Improve the Accuracy, Timeliness, and Relevance of Its Cost-of-Living Measurements
Released: 2020-06-16T00:00:00

Letter

Background BLS Consumer Price Indexes

BLS currently produces a number of different price indexes to estimate price inflation (see table 1). In line with its strategic plan, BLS aims to make these estimates as accurate as possible, meaning that they reflect the average level of price inflation for a selected group of consumers. The accuracy of a price index can be assessed in multiple ways, such as the extent to which the index applies appropriate formulas to data that are complete and drawn from sufficiently large samples covering the relevant group of people. BLS bases its collection of these data on the population covered by the Consumer Price Index for All Urban Consumers (CPI-U). BLS then uses data collected for the CPI-U to produce three other price indexes. After introducing the CPI-U as its primary, or headline, index, BLS maintained a separate data collection for the CPI-W from 1978 through 1980 but found little difference between data for CPI-W and CPI- U. According to BLS, as a result of this and budgetary issues, BLS stopped collecting separate data for the CPI-W in 1981 and began using CPI-U data to derive the CPI-W.

To create the CPI-U, BLS chooses a sample of outlets (e.g., stores or internet sites) at which the CPI-U population shops (see fig. 1 for more information on how BLS creates price indexes). BLS then collects price data at these outlets for goods and services the CPI-U population buys and uses the data to develop basic, or elementary, indexes for each good and service.

BLS combines the elementary indexes into a single, aggregated index by applying a set of expenditure weights—factors that determine, for example, whether a change in the price of apples or mobile phone service has a larger effect on total inflation (see fig. 2). These expenditure weights reflect the proportion of spending consumers direct to each good or service. To develop expenditure weights, BLS directs the Census Bureau to gather data about the relative importance of each purchase within the target population’s “market basket” of consumer goods and services. The Census Bureau collects these data in the Consumer Expenditure Survey, a nationwide household survey conducted by BLS to determine how consumers spend their money that also contains demographic data about the households surveyed. BLS uses 2 years of Consumer Expenditure Survey data to calculate the expenditure weights, in part so the sample sizes are large enough to produce accurate weights.

From data collected to produce the CPI-U, BLS derives two subpopulation indexes—indexes that focus on the spending patterns of a portion of the population of all urban consumers: the CPI-W and the CPI- E. To produce these subpopulation indexes, BLS adjusts the relative importance of price changes in each good and service through a process sometimes referred to as “reweighting,” meaning BLS develops alternate sets of expenditure weights that reflect the spending patterns of the subpopulation. For example, since medical care comprises more of the CPI-E subpopulation’s total expenditures (about 12 percent) than of the CPI-U population’s total expenditures (about 9 percent), the CPI-E gives more weight to medical care than the CPI-U.

BLS also creates a “chained” index using the same data for the entire CPI-U population but changing the formula used to combine indexes for each good and service, known as elementary or basic indexes, into a single aggregated index. This formula captures how consumers shift spending among different types of goods and services as prices change (see text box). In contrast, the other indexes assume that consumers keep purchasing various categories of goods and services in the same proportions over a 2-year period regardless of price changes.

What is a chained price index?

A chained price index uses a formula that is believed by some economists to better approximate a cost-of-living index by more accurately accounting for changes in consumption patterns in response to relative price changes. They contend that such a formula reduces the potential for overstating inflation relative to the other indexes BLS produces, which assume consumers keep buying goods and services in the same proportions no matter their price. Like the other three indexes BLS produces, the Chained CPI-U reflects consumers’ ability to adapt to changing prices by choosing among closely related goods and services as prices change, for example purchasing a different type of apple because it is on sale. However, unlike the other three indexes, the Chained CPI-U further reflects consumers’ ability to choose among all available goods and services as prices change, such as taking a train to work instead of driving when the price of gasoline rises, and purchasing headphones to listen to music during the commute.

We previously reported that, were federal retirement benefits to be indexed to the Chained CPI-U, SSA and other agencies would need to determine whether to base retirement COLAs on final data that may be outdated or preliminary data that may be inaccurate. This is because the data needed to use a superlative index formula only become available after a significant time lag. This lag delays issuance of final monthly estimates for the Chained CPI-U by up to 1 year. Additionally, the chair of a panel convened at the request of BLS to examine issues in measuring the cost of living cautioned that chained indexes may not accurately reflect the way people with varying incomes substitute goods and services. For example, retirees with lower incomes might not have the same ability as retirees with higher incomes to substitute other goods and services when the prices of needed medical care or prescription drugs rise.

BLS receives input on its processes from several sources. For example, BLS receives advice and recommendations from several advisory committees that variously focus on technical issues and the needs of users of BLS statistics. BLS also periodically receives input on its price indexes through external commissions and panels. For example, in May 1995, the U.S. Senate created the Advisory Commission to Study the Consumer Price Index, commonly referred to as the “Boskin Commission,” after its chairman, Michael J. Boskin. In December 1996, the Boskin Commission released its final report identifying sources of bias in the production of CPIs that the commission concluded were causing the indexes to overstate inflation. BLS also receives input on its price indexes through public comment. For example, in May 2019, the Office of Management and Budget issued a request for public comments on the various price indexes produced by BLS and BEA.

Social Security Retirement Benefits

While there are a number of federal retirement benefit programs, Social Security is by far the largest provider of indexed retirement and disability benefits in the United States, paying out over $1,047 billion in retirement and disability benefits in 2019. Social Security was established in 1935 to provide for the general welfare of older Americans by, among other things, establishing a system of federal old-age benefits, including a retirement program. To determine a worker’s initial retirement benefit, Social Security indexes the worker’s earnings to an average wage index. According to SSA, this ensures that a worker’s future benefit reflects the general rise in the standard of living that occurred during his or her working lifetime. Since 1975, Social Security has also indexed retirement benefits after the initial benefit level has been set to a CPI. According to SSA, this ensures that benefits are not eroded by inflation over time. When SSA began indexing benefits, CPI-W was the only national CPI available, and SSA continues to use the CPI-W to determine COLAs.

As we have previously reported, the Social Security program faces financial difficulties that, if not addressed, will affect its long-term stability. In April 2020, SSA projected that Social Security’s retirement program trust fund will be unable to pay full benefits in 2034. We have also reported that, according to projections by SSA and the Congressional Budget Office, use of an alternate index to determine COLAs would have less effect on Social Security’s long-range finances than some other options for addressing the program’s finances, such as changing the taxation of earnings or raising the retirement age. That said, we found that, according to SSA projections, using an alternate CPI to calculate COLAs would affect Social Security’s finances in different ways. Specifically, using the CPI-E would increase expected COLAs and thus program costs and using the Chained CPI-U would decrease expected COLAs and thus program costs, while using the CPI-U would result in little change to either.

National Income and Product Accounts (National Accounts)

Produced by BEA, the National Accounts are a set of statistics on U.S. production, income, consumption, investment, and saving. Among these are Gross Domestic Product, a measure of the goods, services, and structures produced across the economy, and the Personal Consumption Expenditures index, a measure of consumer inflation similar to CPIs, but constructed using different methods and data sources and covering different populations and transactions. Data collected by BEA to produce the National Accounts differ in a number of ways from those collected by BLS to produce CPIs. For example, while CPIs focus on the expenditures of households in urban areas, the National Accounts also include expenditures on institutional populations, such as individuals living in nursing homes. Further, while CPI expenditure data are based on the recollection of consumers, National Accounts expenditure data primarily reflect the records of the businesses that serve consumers. In other words, to collect data on the quantity of goods and services consumed, BLS surveys consumers about how much they bought, whereas BEA surveys companies about how much they sold.

The National Accounts are produced primarily from data collected by federal government agencies. These data include both “statistical” data collected from federal statistical agencies, such as the Census Bureau, as well as “administrative” data collected by federal agencies as a byproduct of administering their programs. For example, BEA uses sample data generated by the Internal Revenue Service in processing tax returns to estimate corporate profits. BEA supplements these statistical and administrative data collected by federal agencies with data obtained from trade associations, businesses, international organizations, and other sources.

BLS Faces Challenges Developing Consumer Price Indexes, but Has Made Limited Use of Data Collected by the Federal Government That May Help It Improve the Indexes’ Accuracy and Timeliness

BLS Faces Challenges Related to the Accuracy and Timeliness of CPIs, Among Others

BLS faces a number of challenges related to the accuracy and timeliness of CPIs, as well as challenges related to measuring inflation for older Americans. Some of these challenges may have implications for federal retirement benefit adjustments.

Accuracy

According to BLS officials and documentation, BLS is unsure if the data sources it uses to produce the CPI-U are adequate to produce accurate subpopulation estimates—specifically, the CPI-E and CPI-W. For the CPI-E, BLS has not evaluated the adequacy of the CPI-U data it uses to measure inflation for the 62-and-older subpopulation. Specifically, BLS has not evaluated the extent to which CPI-U data represent the outlets where members of this older subpopulation shop, the prices they pay, or the mix of goods and services they purchase. BLS considers the CPI-E an experimental index, in part, because of the relatively small sample size within the Consumer Expenditure Survey used to create the expenditure weights for this subpopulation, which account for the mix of goods and services the subpopulation purchases. According to BLS documentation, the expenditure weights for the CPI-U rely on about 65,000 household interviews, which are collected quarterly over 2 years. In contrast, the expenditure weights for subpopulation indexes use about one-third or less of that: 21,000 interviews for the CPI-E and 16,000 for the CPI-W. For the CPI-W, BLS has not evaluated the adequacy of using CPI-U data since 1980, but the relative sample size used to calculate the expenditure weights for the CPI-W subpopulation has been shrinking in part because of declining response rates and demographic shifts away from the occupations included in the CPI-W. For example, occupations in the CPI- W include blue-collar jobs such as clerical, sales, laborer, and construction jobs. BLS officials and documentation indicate that as a result of these demographic shifts and the subsequent shrinking sample size within the Consumer Expenditure Survey, the accuracy of the CPI-W expenditure weights may be deteriorating.

A core element of BLS’s mission is to provide accurate products. Moreover, standards of internal control call for agencies to obtain relevant data from reliable internal and external sources to meet information requirements for meeting their objectives. For BLS, this could include obtaining relevant data from reliable sources for producing CPIs. BLS officials said they have not evaluated the adequacy of the existing data because it is costly to undertake a full evaluation, but there may be cost- efficient ways to do so. BLS also has not evaluated different methods to conduct a cost-efficient analysis. Without taking actions to understand available options for a cost-efficient solution, BLS lacks reasonable assurance that adjustments to Social Security and other retirement benefits are based on indexes that reflect what they are intended to reflect. Specifically, benefits could be subject to adjustment based on potentially inaccurate information.

Most experts we interviewed identified potentially cost-efficient methods to evaluate the adequacy of existing data for subpopulation indexes. For example, five experts we interviewed, including some on BLS advisory groups, suggested that BLS may be able to use existing data to examine the adequacy of using Consumer Expenditure Survey data for the CPI-E. Specifically, one expert suggested that BLS could compare expenditure patterns for the older subpopulation in the Consumer Expenditure Survey to those in third-party data. Another expert added that the overall prices older Americans pay may not be significantly different than the prices the general population pays. For example, gas stations generally charge the same price to each customer regardless of age, so this expert said that it may not be worthwhile for BLS to collect separate price data for older Americans. Another expert indicated that, while it might not be possible to link expenditures and demographics (such as age) for all CPI categories using third-party data, it may be possible for certain categories such as groceries, which are a sizeable portion of the older population’s expenditures. Another suggested that to improve subpopulation indexes, BLS could shift resources from cost savings realized from other ongoing projects.

BLS officials acknowledged some potentially cost-efficient methods could exist to evaluate the adequacy of existing data for subpopulation indexes. For example, they said that a recent change in survey methodology will enable them to connect demographic information with information on where people shop beginning in 2019. The ability to make this connection should allow them to determine whether certain subpopulations shop at the same or different outlets and could help them determine the adequacy of their outlet sample selection. According to agency officials, BLS advisory groups could weigh in on such issues, but BLS has not asked the advisory groups to do so nor do the advisory groups have any recent or ongoing research on indexes for subpopulations such as older Americans. BLS officials added that obtaining transaction and demographic data from credit card companies could help, but cautioned that companies may be unwilling to share these data.

BLS is currently undertaking a project to improve how it estimates its subpopulation indexes, CPI-E and CPI-W, in part by examining changes to the formulas used to apply expenditure weights. As part of its justification for the project, BLS expressed concerns about the decrease in the relative sample size for the CPI-W population in the Consumer Expenditure Survey and reiterated the importance of the CPI-W in adjusting federal retirement benefits. This project is a step in the right direction but does not fully address the question of whether the CPI-U data are adequate to produce CPI-W and CPI-E.

In 2009, BLS began another project to address measurement error in and households’ willingness to respond to the Consumer Expenditure Survey, which is primarily conducted to create expenditure weights for CPIs. According to agency documents, the survey faces increasing costs and declining response rates. One particular goal of the project is to reduce error due to underreporting. For example, BLS is currently testing replacing a paper record of household expenditures with an online form with the goal of more accurately capturing expenditures and maintaining response rates. The project is ongoing and BLS expects to implement changes in stages through and beyond 2022. According to agency officials, the project was not designed to address subpopulation indexes, but instead was designed to address broader issues with the accuracy of the Consumer Expenditure Survey.

Timeliness and Relevance

BLS also faces challenges regarding the timeliness and relevance of CPIs. In particular, most CPIs are published using expenditure data that can be up to 4 years old, and, in this dynamic economy, as expenditure data age, they become less relevant to present-day expenditure patterns. Most of BLS’s price indexes, including the CPI-U, CPI-E, and CPI-W, rely on 2 years of expenditure data and the data require additional time to be collected and processed for use, referred to as a lag. For example, the CPIs produced from January 2014 to December 2015 used expenditure data from 2011 through 2012. BLS officials said reducing the lag could enable more timely use of expenditure data for CPIs but would not be possible without a significant change to the use or design of the Consumer Expenditure Survey.

Another of BLS’s indexes, the Chained CPI-U, aims to incorporate current-period expenditure data, which may be most relevant for current- period price changes, but as we reported in 2019, the data are subject to revision and BLS produces the final, revised Chained CPI-U with a 10 to 12 month delay. BLS officials told us they do not currently have timely enough expenditure data to produce the Chained CPI-U without this delay. We found in our 2019 report that if the Chained CPI-U were to be used to calculate Social Security or other federal retirement benefit COLAs, it could result in permanent differentials stemming from measurement error that would have a larger effect on people who receive benefits longest or have lower incomes.

Other Challenges

BLS also faces several other challenges measuring inflation for older Americans, several of which BLS is examining in the subpopulation project discussed above.

Large purchases. BLS is examining how to treat large purchases that are acquired in one time period but used throughout many time periods, such as owner-occupied housing and durable goods. BLS’s current approach to owner-occupied housing is to calculate what it would cost to rent a similar home. In part, because many seniors own their homes, BLS is considering instead calculating how much it costs to own and occupy the home (e.g., by including mortgage interest payments but not the purchase price of the home).

Definition of average. BLS is also examining whether a subpopulation index should represent the average expenditures of all households (as its CPIs currently do) or the expenditures of an average household. The current approach of representing the average expenditures of all households is simpler because the index can be constructed from information on average expenditures. The alternate approach of representing expenditures of an average household is more complicated because it gives each household equal weight, and requires first constructing a price index for each household, then an averaging of those indexes. According to BLS, the current approach tends to give more relative weight to the purchasing behavior of higher-income households, whereas the alternate approach may be more appropriate for a subpopulation index, such as the one used to adjust Social Security benefits. For example, taking the average of all expenditures tends to reflect the more expensive purchases typically made by higher-income households. In contrast, measuring the average household’s expenditures may better represent expenditures made by a particular subpopulation, such as recipients of federal benefits programs like Social Security.

User needs. BLS is also examining how to define the subpopulation of interest to meet the needs of its users, such as the Social Security Administration. Specifically, CPI-E is based on households headed by someone age 62 or older and the CPI-W is based on households with particular occupations, and BLS is examining whether other definitions could meet user needs. For example, BLS said it plans to contact stakeholders to ask about whether expanding the CPI-W to include all labor force participants (thereby increasing sample size) would meet user needs.

Quality change vs. inflation. A further challenge for all price indexes is determining what portion of the price change is due to changes in quality as opposed to inflation, according to eight of the nine experts we interviewed. BLS has several methods to adjust for quality changes. For example, if an older television is replaced with a new model with an increased price, BLS analysts collect information on the characteristics of those televisions and conduct an analysis to determine how much of the price change is due to a change in quality (e.g., the new television has additional features). The remainder of the price change is attributed to inflation. While accounting for quality change is a challenge for all price indexes, four of the nine experts we interviewed said it may be particularly difficult when measuring inflation for older populations. According to these experts, this is because older populations tend to consume more medical care goods and services, for which quality changes are particularly difficult to measure.

BLS Has Taken Steps to Incorporate Alternative Data Sources into CPIs, but Has Made Limited Use of Other Data Currently Collected by the Federal Government

Alternative data. To improve its price indexes, BLS is exploring the use of alternative data sources, such as “big data” obtained directly from companies, from third parties, or from the internet (see text box below). For example, BLS recently purchased a large private dataset to use in an experimental index for new vehicles. According to BLS, big data may lead to methodological improvements and cost savings in the CPIs. Notably, some big data may provide “real-time” expenditure data that could potentially be used to capture consumer behavior in response to relative price changes, thereby addressing substitution bias. According to agency officials and most experts we spoke with, big data may be promising but incorporating them in the CPIs requires additional considerations and adjustments to the processes BLS currently has in place. For example, the data may not be consistently available with the information needed to produce CPIs. Additionally, big data are not always free and some companies may be reluctant to share these data.

What is “big data?” Big data encompass a number of very large data sets that can be structured or unstructured and have the potential to be mined for information. Web-scraped data and scanner data are two prominent types of big data relevant for consumer price indexes. Web- scraped data are price data collected on goods sold online. Scanner data include price and quantity data on sales of goods obtained by scanning bar codes for goods, such as at electronic points of sale in retail outlets. Advances in technology have allowed large amounts of data to be collected and stored easily and could be used in consumer price index construction.

In addition to big data, BLS currently uses some administrative data collected by the federal government to improve inflation estimates for certain goods and services. For example, BLS obtains information from the Department of Energy on household consumption averages for electricity and piped gas service. It also uses administrative data from the Centers for Medicare & Medicaid Services about which facilities provide adult home care. According to BLS officials, they are unable to use some administrative data (e.g., certain federal tax data) because of current law.

Other data collected by the federal government (National Accounts data). While BLS is exploring numerous alternative data sources, BLS has not fully explored the potential to update expenditure weights on a more frequent basis using supplementary data from the National Accounts in years when the most current biennial weights using Consumer Expenditure Survey data are not available. As discussed earlier, BLS typically requires 2 years of data from the Consumer Expenditure Survey to produce expenditure weights, which have a lag. In contrast, National Accounts data comprise administrative and statistical data representing the whole economy, many of which have a large sample size and are available on an annual basis. Standards of internal control call for agencies to obtain relevant data from reliable internal and external sources in a timely manner to meet information requirements for meeting their objectives. For BLS, this could include obtaining relevant data from reliable sources for producing CPIs. As part of its strategic plan, BLS maintains goals to improve the accuracy and timeliness of BLS data and to ensure relevance in an ever-changing economy. Without adequately exploring the potential of using National Accounts data to supplement Consumer Expenditure Survey data, BLS may be missing an opportunity to move closer towards those goals. Over time, expenditure survey data lose their accuracy and relevance to the present-day expenditure patterns of consumers, which can introduce bias in measures of inflation used to adjust federal retirement benefits. For example, the longer the time period between expenditure weight updates, the longer the delay to include new products in the expenditure patterns reflected in the CPIs. This delay could become increasingly important because of the rapid development in new technology, such as smart phones.

Of the 15 publications we reviewed, six discussed ways to improve the CPI and four of these suggested more timely expenditure weight updates could make the CPIs more accurate and relevant. For example, a 2009 working paper by BLS staff found that more frequent weighting may offer better representation of current price change, as well as a closer approximation to a cost-of-living index. In particular, the authors simulated updating expenditure weights annually, which resulted in slower inflation increases that the authors posited are a closer approximation to a cost-of-living index. While these improvements may not be currently possible given the lag in Consumer Expenditure Survey data, the authors conclude that further examination of the weighting issue is a potentially fruitful avenue of research. The three other studies similarly indicated that more timely weight updates would result in more relevant CPIs, for example by better reflecting changes in consumer spending patterns.

BLS officials acknowledged that updating the weights more frequently would make the index more relevant, though they did not believe using the Consumer Expenditure Survey to do so was practical in part because they said it would require additional costs to increase the sample size. In 2002, BLS increased the frequency of its weight updates from every 10 years to every 2 years, which they said was an improvement but required a sample size increase in Consumer Expenditure Survey. As previously described, the Consumer Expenditure Survey faces increasing costs and declining response rates and, according to agency officials, obtaining a large enough sample to update weights annually would require a 50 to 100 percent increase in sample size, for example, to avoid an increase in sampling error. Indeed, three studies we reviewed suggested that it can be challenging to obtain enough responses for household surveys such as the Consumer Expenditure Survey, indicating that alternate data sources may become more important.

In contrast, BLS officials acknowledged that National Accounts data could provide useful supplementary information if the expenditure survey is not providing timely enough data. However, BLS officials said they have not explored using National Accounts data, in part because they have not examined the effects of altering the expenditure weights in about 10 years. BLS officials expressed concern that National Accounts data can be subject to revision. According to the Bureau of Economic Analysis (BEA), the revisions do not reflect errors but are driven by the incorporation of more complete source data. BLS officials also noted that some National Accounts data are adjusted by the CPI, so BLS would have to remove the CPI’s effect in order to use National Accounts data in the CPI.

Moreover, the supplementary use of National Accounts data could also help address some of the concerns with measurement error in household surveys, according to some literature we reviewed. Specifically, National Accounts data could be used to address underreporting due to recall bias, the difficulty some survey respondents have recalling infrequent purchases, or underreporting of certain goods that may be seen as socially undesirable, such as tobacco and alcohol. For example, according to a recent Brookings Institution report, the National Accounts data used for the BEA’s Personal Consumption Expenditure index weights are mostly based on business surveys and administrative data and thereby avoid the reporting biases inherent in the Consumer Expenditure Survey. BLS’s Technical Advisory Committee recommended using administrative data to address such underreporting in fiscal year 2016, as did a National Academy of Sciences report in 2013. While BLS has taken steps toward increased use of administrative data, BLS has not fully implemented the Technical Advisory Committee recommendation as of March 2020.

Selected Countries Use Various Strategies, Such As Obtaining Data from Alternative Sources and Bolstering Collaboration with Stakeholders, to Update Their Indexes for Retirement Benefits

Use of Retiree-Specific and Chained Price Indexes for Adjusting National Pension Benefits Is Relatively Uncommon

Our review of Organisation for Economic Co-operation and Development (OECD) countries’ national pension systems revealed that it is relatively uncommon to use a retiree-specific index (i.e., a CPI for the older subpopulation) for the purpose of adjusting national pension benefits. Of the 36 OECD countries, 27 have national pension programs in which indexation is based, at least in part, on prices after initial benefits have been set, similar to Social Security in the United States (see app. I). Most OECD countries use their primary measures of inflation to adjust national pension benefits, according to reports and documents about the retirement systems in these countries. Of the 27 countries using prices to adjust national pension benefits, we found evidence in 10 that the national statistical agency produces an index for the older subpopulation. Each of these 10 countries generally uses the same price information for the older subpopulation index as the main CPI but reweights the price information based on the expenditures for that subpopulation, rather than gathering new information that is unique to that group (see text box). A similar approach is used for the CPI-E in the United States. However, of these 10 countries, only four countries use the index for the older subpopulation to adjust their national pension benefits (Australia, Czech Republic, Hungary, and the Slovak Republic). The others produce the subpopulation index for research or other purposes, but do not use it for pension benefit adjustments.

Agency officials in all three of our case study countries (Australia, New Zealand, and the United Kingdom) said they generally saw a value in having a primary index for macroeconomic purposes, such as inflation targeting, and a subpopulation index that could be used for other purposes, such as indexation of benefits.

Methods for Validating Use of Existing CPI Data in Subpopulation CPIs In the three case study countries we selected for review, each national statistical agency relied upon different approaches to validate the use of existing data from the primary (main) CPI in the subpopulation CPI. Agency officials indicated that some of the methods for validating the use of existing CPI data for the subpopulation CPIs were cost efficient.

Australia agency officials said they validated the use of existing data in the index for the older subpopulation in part by both researching whether pensioners pay different prices or shop at different outlets and cross-checking some data from industry sources. Officials said they expected that pensioners and the general population generally pay the same prices for most items and included different prices in the index for the older subpopulation for those items known to be discounted for pensioners.

To get a better sense of the older population’s expenditures, they also increased the sample size of the expenditure survey from about 7,000 households to about 10,000 households to include more pensioners.

New Zealand agency officials said they validated the use of existing data in part by using existing expenditure data to confirm that goods and services most important to the older subpopulation were adequately represented in the data. They also said they consider the coverage of the subpopulation group when determining the make-up of the CPI basket. Since older people may shop at different stores than the general population, New Zealand’s statistical agency also developed separate outlet weights for the older subpopulation, which more accurately reflect the different mix of outlets, or stores, frequented by this group, according to agency officials. Overall, officials said they found that using subpopulation-specific outlet data instead of general CPI outlet data had very little impact on the index for the older subpopulation.

United Kingdom agency officials said they validated the use of existing data by organizing expenditure data from the household survey into categories that align with national expenditure data, which allowed them to generate bigger samples than exist in the household survey data. As a result of the larger sample, their statistical agency said they were able to achieve more precise estimates for the index for the older subpopulation.

It is also relatively uncommon for a country to produce a chained index for the purpose of adjusting national pension benefits. Of those 27 OECD countries that are using price indexation, five of them produce a chained index (Australia, Canada, the United States, the United Kingdom, and New Zealand). However, none of the OECD countries use the chained index to adjust their national pension benefits. In our three case study countries, the statistical agencies used the chained index as an analytical tool to measure bias in the CPI or for comparative purposes. Officials we spoke with said that the delay required to produce a chained index made it impractical to use the index to adjust benefits. While some of the stakeholders in selected case study countries indicated it could be theoretically possible to create a chained CPI for the older subpopulation, we did not identify any countries with such an index during this review.

Selected Countries Are Supplementing CPI Data with Other Government- Collected Data to Help Bolster Gaps in Information

While government-collected data are often collected for reasons other than the production of the CPIs, the three selected case study countries are using government-collected data to help fill the gaps in data they collect expressly for the CPI (see table 2).

According to agency officials in the three selected countries, use of this government-collected data improves accuracy of the CPIs and can be a relatively affordable way to supplement data collected for the CPI.

National Accounts, key sources of government-collected data, are typically used for national summary measures like the Gross Domestic Product. However, all three of the selected countries are also using relevant consumption data from National Accounts to supplement their CPI data, which agency officials in Australia said is in-line with recommendations from the International Labour Organization (see text box). Australia, New Zealand, and the United Kingdom are all using their National Accounts data to supplement expenditure survey data in their CPIs, while New Zealand is also using another form of government- collected administrative data to improve its CPIs.

International Guidance for Calculating CPIs and Subpopulation Indexes The International Labour Organization produces a manual that provides an overview of issues that national statistical offices can consider when making decisions on how to deal with the various problems in the compilation of Consumer Price Indexes. Researchers from many countries’ national statistical agencies, universities, and international organizations (such as the World Bank, International Monetary Fund, and Organisation for Economic Co-operation and Development) are involved in creating the manual. The manual also establishes international conventions, such as a suggestion that countries regularly evaluate the use of average wages as opposed to price indexes (and vice versa). Last published in 2004, an update to the manual is scheduled to be released in 2020. The upcoming revised manual is expected to elaborate on the use of National Accounts data and alternative data sources to develop expenditure weights.

Australia. Australia’s statistical agency uses consumption data from their National Accounts to update the CPI expenditure weights more frequently than officials said was previously possible. Using this data has helped reduce substitution bias, meaning that the data better reflect changes in consumer purchases in response to price changes. Previously, Australia updated its expenditure weights every 6 years, when its household expenditure survey was released. In other words, the CPI was previously calculated assuming that consumers’ expenditure patterns did not change for 6 years. As a result, the CPI did not account for substitution patterns to different goods and services over significant periods of time, leading to bias in the CPI. In 2018, the Australian statistical agency incorporated National Accounts data in the CPI in those years when the expenditure survey was not conducted, allowing the expenditure weights to be updated annually to reflect what statistical agency officials described as more timely and relevant consumption patterns and to improve the accuracy of the data. According to Australian statistical agency officials, they did not have the budget to increase the frequency of their household expenditure survey, which they said is very costly. Instead, officials said they researched alternative ways that would allow for more frequent reweighting and settled on using the National Accounts data in between survey years to update the weights annually. This approach does not require a budget increase because the National Accounts data are already produced. Australian officials said more frequent weighting helped reduce substitution bias in their CPIs by about 0.2 percentage points per year, which can have a large impact on benefits over time.

By incorporating consumption data from the National Accounts, Australian statistical agency officials said they can generate more timely and relevant CPI measures, including the subpopulation indexes. Australia’s index for the older subpopulation, called the Pensioner and Beneficiary Living Cost Index, also benefits from more frequent updates of the expenditure weights and subsequent reduction in substitution bias in the CPI, according to agency officials. Agency officials said that despite not having demographic information in the National Accounts, their methods have made use of this consumption data fit for purpose for the subpopulation indexes, and the subpopulation indexes are as methodologically sound as the primary CPI.

New Zealand. New Zealand’s statistical agency also uses National Accounts data to estimate expenditure weights for insurance services, which are relatively difficult to measure in survey data, according to agency officials. Specifically, the expenditure weights for health and life insurance are based on data from the National Accounts.

United Kingdom. In the United Kingdom, annual spending data from the National Accounts are the main source for CPI expenditure weights, as stakeholders noted that the National Accounts spending data are more precise and timely than their household expenditure survey. According to statistical agency officials, household expenditure data are ultimately obtained by organizing the United Kingdom’s expenditure survey data into categories that align with the National Accounts and scaling up these data to the National Accounts data. Officials said this method allows the United Kingdom’s statistical agency to achieve larger sample sizes, and thus smaller variances and more precision in estimates for subgroup indexes. United Kingdom officials said that their National Accounts estimates are more accurate and comprehensive than their household expenditure survey, which has a smaller sample size of nearly 6,000 households. Having more accurate expenditure data and weights leads to a more accurate and relevant primary index for pension benefits, as well as a more accurate subpopulation index, according to agency officials. The National Accounts data also help the United Kingdom adjust for any potential underreporting of particular goods in the household expenditure survey, such as alcohol, further increasing the accuracy and relevance of the dataset, according to officials. Collecting prices directly from the source is more accurate than relying on someone to recall how much they spent on items, according to one stakeholder.

Government agencies from selected countries also produce other administrative data that can be useful in measuring the CPI. For example, New Zealand’s statistical agency partnered with the Ministry for Business, Innovation, and Employment to use its tenancy bond database, which covers approximately 85 percent of all rental housing units in the country. These data facilitated a new way to measure rent in their CPI. Moreover, this partnership enabled New Zealand’s statistical agency to create an index of rent prices monthly, instead of quarterly, which resulted in a more accurate and timely depiction of what people are spending on rent and a more accurate indexation of benefits overall. According to agency officials, the transition to these administrative data replaced the CPI survey of landlords, and in doing so it lowered respondent burden, increased the timeliness of the rental component of New Zealand’s CPI, and improved population coverage. In all of our case study countries, various data are used to measure housing prices (see text box).

Housing and the Consumer Price Index Measuring the change in housing prices for CPI is widely acknowledged by experts to pose methodological and data challenges. In response, national statistical agencies have developed a variety of approaches to address the measurement of owner-occupied housing costs, both in the primary CPI and subpopulation indexes. Officials in the national statistical offices of the case study countries said that one of the factors underlying the approach to housing is whether the measure should reflect inflation in the economy overall or inflation as experienced by households.

In Australia and New Zealand, the primary CPI includes price changes stemming from the purchase of a new home but not via mortgage interest payments (known as the acquisitions approach), while the subpopulation index excludes the purchase of a new home but includes mortgage interest (referred to as outlays or payment approach).

In the United Kingdom, there are two versions of the primary CPI: one that uses “rental equivalence” (a calculation of what the owner would pay in rent for an equivalent house) and one that excludes owner-occupied housing costs. In addition, the United Kingdom’s subpopulation index uses a payments approach.

Selected Countries Are Using Alternative Big Data Sources to Get More Data in a More Timely Way

Officials in our selected case study countries said they are using alternative big data sources, such as web-scraping data and transactional (scanner) data to help them more accurately index their national pension benefits (see table 3).

These officials said that these alternative data sources allow countries to obtain a higher volume of data and more accurate data to incorporate into their CPIs, subsequently making the indexation of benefits more accurate. Electronic price data obtained from a retailer, whether through the retailer’s website or through scanner data the retailer shares with the national statistical agency, reflects accurate and timely data on the price and quantity of goods and services sold. Electronic price data can be an improvement over data collected in household expenditure surveys, for example, as several experts and agency officials in one case study country noted that household expenditure surveys suffer from recall bias, resulting in less accurate spending data. The three selected countries are at different stages of incorporating scanner data into their CPI. Officials at the national statistical agencies in all three of our case-study countries stated that they are primarily focused on incorporating scanner data from grocery stores into their CPI. Using grocery store data is possible, in part, because these countries contain a relatively small number of stores that dominate grocery sales, according to agency officials, which is a difference from the United States.

Australia. According to stakeholders, the Australian statistical agency developed a formula that incorporates a chained formula into a portion of the CPI using high- frequency scanner data from the country’s dominant grocery stores, which provides timely price and expenditure data on food items for their indexes. Integrating this type of high- frequency data is not easy, they said, since the traditional CPI formulas are not built to handle the volume of data that scanner data produce. However, in consultation with academics and statistical agencies from around the world, Australia was able to develop a chained formula that uses an innovative statistical method, known as a multilateral approach, to incorporate the scanner data. As a result, the portion of the CPI for which Australia has scanner data (about one-sixth of the CPI, comprised mostly of food and other grocery data) is based on a chained formula. Incorporating these data allows the country to include all of the products available in the datasets, rather than a small sample of products, leading to a more accurate calculation of food prices and a more accurate index overall, for both the general population and the older subpopulation, according to agency officials. Stakeholders in Australia noted that the international price statistics community has since reached a consensus that multilateral methods are the most effective way to capitalize the full amount of information provided in scanner data, and they said that the forthcoming update of the International Labour Organization’s CPI Manual is expected to recommend this method as well.

New Zealand. New Zealand’s statistical agency is working towards incorporating more scanner data, primarily from its two large supermarket chains, in the production of the country’s CPIs, which will help achieve a more accurate index for both the general population and the older subpopulation, according to agency officials. New Zealand started using retail scanner data to supplement its expenditure data in its CPI in 2006, and in 2014 New Zealand incorporated direct measurement from scanner data for consumer electronics products into its CPI. Officials from the national statistical agency said they hope to expand their use of this type of big data in the near future. They have already received the data from supermarkets, whose goods account for roughly 20 percent of the goods and services in the CPI, but they have not yet integrated the data into their CPIs. Agency officials said they expect to integrate this in the next year. New Zealand’s statistical agency officials said they have a goal to obtain scanner data for other CPI components soon as well, such as fuel.

United Kingdom. In the United Kingdom, agency officials said improvements in technologies have resulted in new alternative sources for price data that could be used in the compilation of their price indexes in the near future. The United Kingdom’s statistical agency is currently exploring both scanner data and online price data. The agency currently has several streams of research looking into the expanded use of alternative data, including research studying the feasibility of moving away from collecting prices manually towards using electronic means wherever feasible and efficient. The agency is now receiving web-scraped data from an online source that captures prices from online sales of goods like clothing. The United Kingdom’s statistical agency is also continuing to engage with retailers on receiving scanner data covering areas such as clothing and groceries, targeting some of the largest retailers from which the agency currently manually collects prices. These data sources may provide a more efficient way to capture the increase in online expenditures that has occurred over the last decade, and will likely continue to occur. These new data are initially being used for research work, but over time the web-scraped online prices and scanner data will be used when calculating primary inflation indexes, according to agency officials. The research done by the United Kingdom’s statistical agency into grocery store items has also enabled officials there to explore different methods of collecting web scraped prices in-house. The officials said this has led to wider benefits for the agency in general, with an increase in knowledge and experience that has contributed to the success of other big data projects.

Selected Countries Collaborate and Consult with National Stakeholders and Experts When Implementing Changes to Their CPIs

Our selected case-study countries use committees with stakeholders and advisory panels, including academic researchers with subject matter expertise, to implement innovative changes to their CPIs (see table 4).

The statistical agencies in these three countries have shown a willingness to act on recommendations that came out of these collaborative efforts. These countries are also seeking input from the international statistical community, which country officials said has led to positive developments in their CPIs.

Australia. Australia’s statistical agency has taken a variety of approaches to collaborate with external stakeholders, which agency officials said has led to positive changes to their CPIs, and thus indexation of benefits over the years. According to agency officials, Australia’s collaborative efforts include: conducting regular reviews and seeking stakeholder input every 6 years with the release of the expenditure survey; convening workshops with stakeholders including both academics and users (e.g., the agencies that distribute benefits); participating in international conferences to receive feedback on changes to the country’s CPI and subpopulation indexes; partnering with methodology experts in other agencies such as the Treasury and central bank, occasionally by obtaining staff on detail; and commissioning reports that research and review measures to strengthen the financial security of seniors.

These reviews and associated collaborative efforts have helped the agency learn more about the issues it faces and have helped trigger changes that will improve the accuracy of the nation’s CPI, according to agency officials. For example, as discussed above, agency officials said that a 2011 CPI review revealed concern by the Reserve Bank of Australia and others that the infrequent reweighting was resulting in bias in the CPI that affected inflation targeting by the central bank, as well as benefit expenditures. This review helped spur innovations, such as including the incorporation of scanner data into the nation’s CPI, which delivered positive results with respect to more timely and relevant data being used to estimate inflation. Australia’s statistical agency officials said they sought extensive input from key governmental stakeholders, a number of academic experts, as well as international experts to research how to best incorporate scanner data into their CPI, which agency officials noted was necessary to facilitate the integration of high-frequency scanner data into the CPI. They also conducted numerous bilateral and multilateral consultations with key stakeholders in the government that use CPI data, including the Reserve Bank of Australia, the Treasury, Department of Finance, Department of Social Services, and State Treasuries. Australian statistical agency officials suggested that consulting with users of the data frequently was an important part of implementing changes to the measurement of the CPI and subpopulation indexes.

New Zealand. New Zealand’s statistical agency has also used CPI advisory committees composed primarily of external stakeholders who make use of the agency’s CPIs. For example, in 2013 New Zealand’s statistical agency convened a committee to independently review the methods and practices used to compile the CPI and make recommendations, for example, about how additional indexes should be measured. The committee also incorporated public submissions on the scope and uses of the CPI, for example, from nongovernmental organizations and interest groups such as retiree advocacy groups. The committee then released a report recommending the creation of additional CPIs that are designed for microeconomic purposes, such as the indexation of retirement benefits, to better reflect changes in the purchasing power of the incomes of particular subgroups of the population, like the older subpopulation. The committee also recommended that New Zealand’s statistical agency review the sample size and collection methods of their expenditure survey to improve the reliability of expenditure estimates of the required population subgroups so that the estimates could eventually be of high enough quality to be published, which they subsequently were. According to officials, the committee’s report helped lead to the creation of New Zealand’s subpopulation indexes. Moreover, the committee recommended that the statistical agency try to use retail scanner data to measure price change and stated that the method aligns with international best practices. New Zealand’s statistical agency recognized these best practices and the international consensus that multilateral methods are the preferred way to incorporate big data. Indeed, it has started to use these methods in the rental prices data and it plans to continue to research implementing these methods further.

United Kingdom. The United Kingdom has also developed advisory panels on consumer prices to provide independent advice to the National Statistician, which officials said has allowed the United Kingdom’s statistical agency to learn more about challenges with the nation’s CPIs and to find possible solutions. Similar to the United States, the United Kingdom has advisory groups on technical issues, as well as on the uses of price indexes. The reports published by various advisory groups have raised technical issues with the Retail Price Index (RPI), which is the United Kingdom’s longest running measure of inflation. These technical issues resulted in the RPI being higher than the CPI. Ultimately, agency officials said consultations and advisory panel input helped lead to the RPI being decertified as a national statistic (see text box). The United Kingdom’s statistical agency also hosted numerous meetings and a collaborative workshop about the conceptual foundations of its subpopulation indexes, which are currently being developed. According to agency officials, obtaining input from internal and external stakeholders has been critical to developing solutions to indexation challenges.

The United Kingdom’s Experience Changing Price Index Used for Pension Adjustments Changing the index used for benefit adjustments can be difficult, as switching price indexes can involve tradeoffs. For example, public and private pension benefits in the United Kingdom have traditionally been indexed by the Retail Price Index (RPI), the oldest index in the United Kingdom. The United Kingdom recently switched indexation of certain government benefits, including pension benefits, from the RPI to the slower-growing CPI. This is expected to result in lower payouts from the government. In contrast, the government continued using the faster- growing RPI for some provisions, such as student loan interest rates, that resulted in higher payments to the government. Stakeholders suggested that having multiple measures of inflation can create incentives for the government to use different indexes for its own budgetary advantage, with pensioners receiving lower benefit adjustments and students facing relatively higher loan payment adjustments. The United Kingdom’s experience highlights that changing the index for benefits may result in advantages and disadvantages for different groups and thus may be politically difficult, according to agency officials.

Conclusions

Federal retirement programs like Social Security have relied upon a subpopulation price index to adjust benefits since automatic cost-of-living adjustments were first enacted almost 45 years ago. This index estimates changes in purchasing power for wage earners as opposed to changes in the standard of living or some other type of measurement. In recent years, numerous legislative proposals have been suggested to change this index from one that measures the purchasing power of wage earners to one that targets some different population, for example one solely focused on the elderly. Much of the debate over using a different index has centered on the ability (i.e., the accuracy) of the indexes to capture changes in the cost of living for a particular group in society.

BLS is unsure whether the data sources it currently uses are adequate to produce accurate CPI-E and CPI-W subpopulation indexes on a timely basis, according to BLS officials and documentation. While the CPI-E is experimental and not used by federal programs, the CPI-W is used to adjust billions of dollars of Social Security and other federal retirement program benefits. It is therefore critical that the measurement be as accurate as possible. However, ensuring the measurement’s accuracy may require a reexamination of the underlying data used to produce the subpopulation indexes. BLS has not evaluated the adequacy of existing data because it is costly to undertake a full evaluation, according to agency officials. But experts we interviewed, including some on BLS advisory groups, indicate there may be cost-efficient ways to conduct such a review. Although the experiences of other countries may not be directly applicable, other countries have found ways to evaluate the use of existing data for their subpopulation indexes, and officials in all three of our case study countries expressed the view that some of these methods were cost efficient. Absent BLS evaluating the adequacy of the existing data it uses to produce its subpopulation indexes, BLS will continue to be uncertain if its subpopulation indexes are accurate and it may not learn of potential areas for improvement.

In addition, BLS currently relies on the Consumer Expenditure Survey to produce expenditure weights that measure the mix of goods and services consumers purchase and, because of survey shortcomings and processing lags, the weights reflect spending patterns that can be up to 4 years out of date. Although BLS has taken other steps to improve the accuracy, timeliness, and relevance of data used in the CPIs, BLS has not fully explored the potential to update expenditure weights on a more frequent basis using annual data from the National Accounts, which are currently collected in part to measure Gross Domestic Product. While not specifically designed for use in CPIs, the National Accounts data may provide BLS an opportunity to supplement Consumer Expenditure Survey data in the intervening years. Moreover, some literature we reviewed indicated that the use of National Accounts data has the potential to mitigate measurement error in the Consumer Expenditure Survey, thereby increasing accuracy. Without adequately exploring the potential of such an option, BLS may be missing an opportunity to improve its CPIs.

Recommendations for Executive Action

We are making the following two recommendations to the Department of Labor:

The Secretary of Labor should ensure that BLS explores cost-efficient ways to evaluate the data sources currently used to produce subpopulation indexes, such as by engaging more directly with other stakeholders or seeking input from its advisory groups and other knowledgeable entities about approaches to expand data collection in a cost-efficient manner. (Recommendation 1)

The Secretary of Labor should ensure that BLS explores the use of already collected National Accounts data to produce more accurate, timely, and relevant CPIs. (Recommendation 2)

Agency Comments and Our Evaluation

We provided a draft of the report to the Department of Labor, the Social Security Administration, and the Department of State for their review and comment. We also sent an informational copy to the Bureau of Economic Analysis. The Department of Labor and the Social Security Administration provided technical comments, which we have incorporated where appropriate. In an email, the Department of State said it had no comments on the report. The Department of Labor also provided written comments, which are reproduced in appendix III and discussed below.

In its written comments, the Department of Labor stated that BLS continually improves its measures according to a guiding principle to provide accurate, objective, relevant, timely, and accessible information. The Department of Labor agreed with the first recommendation to explore cost-efficient ways to evaluate the data sources currently used to produce subpopulation indexes and stated that it would continue to investigate improvements to subpopulation indexes.

The Department of Labor disagreed with the second recommendation to explore the use of National Accounts data in the construction of its indexes, stating that the National Accounts data are not a replacement for Consumer Expenditure Survey data. While we agree that the National Accounts data are not a wholesale replacement for the Consumer Expenditure Survey data, we believe that it would be useful to examine National Accounts data as an augmenting, alternative source of data that could supplement or enrich the Consumer Expenditure Survey. Such an effort could potentially lead to more accurate, timely, and relevant CPIs. Although the Department of Labor stated that the Consumer Expenditure Survey is a continuous survey and that data are received quarterly, most CPIs still rely on expenditure weights based on Consumer Expenditure Survey data that are up to 4 years out-of-date. In addition, the Consumer Expenditure Survey faces increasing costs and declining response rates.

The Department of Labor stated in its comments that it is exploring ways to accelerate the data collection and processing time and that it periodically investigates the frequency of updating expenditure weights. We commend the Department of Labor for considering these efforts, and we maintain that they could take further action to explore additional opportunities for improvement. For example, the Department of Labor could research the extent to which there are instances or categories for which the National Accounts data could be used to produce more up-to- date expenditure weights than the Consumer Expenditure Survey. As we noted in our report, Department of Labor officials told us they periodically examine National Accounts expenditure data to explore differences with the Consumer Expenditure Survey data, not to explore supplementary use of alternative data. While it cannot be ensured that every expenditure data point in the National Accounts will be of use for producing CPIs, we maintain that further exploring the National Accounts expenditure data as a complement to the Consumer Expenditure Survey data may provide opportunities for BLS to improve the accuracy, timeliness, and relevance of its CPIs.

We are sending copies of this report to the Secretary of Labor, the Commissioner of Social Security, and the Secretary of State. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: National Pension Indexation Formulas in the 36 OECD Countries

Appendix II: Additional Information about Selected Case Study Countries

Appendix II: Additional Information about Selected Case Study Countries technical corrections as necessary. We note also that the fact that a legal feature was successful in one or more of the countries we visited, which may have significantly different cultures, histories, and legal systems than the United States, does not necessarily indicate that it would be successful in the United States.

Appendix III: Comments from the Department of Labor

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact Staff Acknowledgments

Charles A. Jeszeck, (202) 512-7215 or jeszeckc@gao.gov In addition to the contact named above, Michael Collins (Assistant Director), Laura Hoffrey (Analyst in Charge), Emilio Fonseca, Kathleen McQueeney, Tom Moscovitch, and Julie Miller made key contributions to this report. Also contributing to this report were Deborah Bland, Alicia Cackley, Charles Ford, Sarah Gilliland, Susan Irving, Kelsey Kreider, Sheila McCoy, Jessica Orr, Oliver Richard, Joseph Silvestri, Almeta Spencer, Curtia Taylor, Frank Todisco, Walter Vance, Adam Wendel, and Sirin Yaemsiri.