BLS WORKING PAPERS

U.S. Department of Labor

U.S. Bureau of Labor Statistics

Office of Prices and Living Conditions

The Problem with Normalizing Preferences that

Change in a Cost-of-Living Index

Gregory Kurtzon, U.S. Bureau of Labor Statistics

Working Paper 534

November 2020
The Problem with Normalizing Preferences that

Change in a Cost-of-Living Index

Gregory Kurtzon U.S. Bureau of Labor Statistics

Division of Price and Index Number Research

2 Massachusetts Ave. NE Room 3105

Washington, DC 20212

November 2020

Abstract

There are new proposals for prices indexes that attempt to correct

for what they consider bias in standard indexes from changes in consumer preferences. But these proposals have a fundamental problem that

changes in preferences between two periods cannot be identiÖed by data

on prices and quantities with only a normalization. This paper shows

that the required normalization is not free, so that an arbitrary choice of

normalization can yield any desired index result. In fact, a normalization

using the Sato-Vartia weights yields a Sato-Vartia index, implying exactly

zero bias.

1 Introduction

Multiple prices indexes that incorporate changing consumer preferences

have been proposed. Such studies claim that standard cost-of-living indexes

(COLIs) su§er from bias because they are deÖned for only one set of preferences.

These studies include Redding & Weinstein (2020) (RW), Ueda, Watanabe, and

Watanabe (2019), and Gabor-Toth & Vermeulen (2020). But these proposals

have a fundamental problem that changes in preferences between two periods

cannot be identiÖed by data on prices and quantities. At any given time, the

data can only identify relative preferences over goods in that time period, not

how the entire standard of living (utility) compares between periods. Therefore,

they use a normalization to Öx the level. Using the RW framework, this paper

shows that the required normalization is not free, so that an arbitrary choice

among di§erent normalizations will yield almost any desired index result.

For illustration, consider if there were only two goods, apples and oranges. In period s < r, apples are preferred more, such that if the prices were

I would like to thank Robert Martin and Peter Zadrozny for suggestions and comments.

All estimates and analyses based on Nielsen data are by the author and not the data provider.

1
equal, the consumer would buy more apples, and vice versa in period r. The proposals that use only price and quantity data cannot equate preferences between

the two periods without normalizing a set of preference parameters. Consider if

consumers simply enjoy both goods proportionately more in the second period,

so that consuming the same quantities gives a higher level of the standard of

living. Then the cost of obtaining the standard of living of the Örst period would

be lower, and there would be no way to rule out this possibility with the data

because the observed shares would be the same. Therefore, the preferences are

normalized to Öx the level, such as by Öxing some mean of the parameters, such

as the mean of the desire for apples and oranges. Putting aside that this still

doesnít identify the level change, any di§erent weighting of that mean is still

consistent with the data but could yield almost any relevant result. So if the

desire for apples was averaged with double the desire for oranges and normalized to the same number, the observed shares of apples vs. oranges would be

the same but the consumer would be better o§ in r relative to s for the same

bundle. Since a COLI measures the change in cost of a standard of living, the

COLI would be lower.

Section 2 presents the issue formally using the RW framework as the

primary example, and the identiÖcation problem with the normalization, and

shows alternative normalizations. Section 3 presents the calculated indexes implied by the alternative normalizations and compares them. Section 4 concludes.

2 Indexes with Changing Preferences

RW models consumer preferences with constant elasticity of substitution

utility, which are homothetic, so the expenditure function can be characterized

by the unit expenditure function1

P



t =

"

k2
t1;t 

pkt

kt 1

# 1

1

(1)

where 
t1;t denotes the set of all goods common to both periods t-1 and t,

pkt is the price of good k in period t,  is the elasticity of substitution between

goods, and kt is a preference parameter for good k in period t, which could be

interpreted as good quality, appeal, as a demand shifter, or speciÖcation error

or just a demand residual. While Martin (2020)2shows that how the common

goods are deÖned in this framework changes the results, here I abstract from

the choice of common goods.

1 This is RW equation (1).

2 This is shown in Appendix C of Martin (2020).

2
The COLI for CES preferences is therefore

COLIt1;t =

P



t

P



t1

=



k2
t1;t 

pkt

kt 1

 1

1



k2
t1;t 

pk;t1

kt 1

 1

1

(2)

The proposed RW index is,

CCVt1;t =



k2
t1;t 

pkt

kt 1

 1

1



k2
t1;t 

pk;t1

k;t1

1

 1

1

(3)

where CCV denotes the price index for the goods that are common to both

periods3. The Sato-Vartia (SV) index is an exact COLI for CES preferences4,

given by



SV

t = k2
t1;t 

pkt

pk;t1

!



kt

(4)

where !



kt are the Sato-Vartia weights, !



kt =

s



kts



k;t1

ln s

ktln sk;t1

k2
t1;t

s

ktsk;t1

ln s

ktln sk;t1

. The "taste

shock bias" given by the CCV5is therefore

ln CCV

t = ln SV

t 



taste-shock bias

k2
t1;t!



kt ln kt

k;t1



(5)

.

The goodsíshares in CES are6

s



kt =



pkt

kt 1

k2
t1;t 

pkt

kt 1

(6)

which implies7

(P



t

)

1 =

1

s



kt



pkt

kt 1

(7)

.

3 This is equivalent to equation (9) in RW.

4

If preferences change, the SV index is exact for preferences that are intermediate between

the two periods. See Feenstra & Reinsdorf (2007).

5 This is RW equation (13).

6 This is RW equation (2).

7 This is equivalent to RW equation (2).

3
Because all the kt parameters could be divided by a constant and yield

the same shares and index, only the ratios kt

jt

are identiÖed. Therefore, for

an index for periods t-1 to t, they are normalized by the unweighted geometric

mean of all kt, so that

k2
t1;t

1

Nt1;t

kt  1 (8)

, where Nt1;t denotes the total number of goods common between period

t-1 and t.

Consider alternative normalizations with weights wkt, all equally consistent with the data in (6),

k2
t1;t

wkt

kt  1 (9)

k2
t1;twkt = 1 (10)

. Taking the log of both sides of (7), time di§erencing and rearranging yields

(1  ) ln P



t

P



t1

= (1  ) ln pkt

pk;t1

 (1  ) ln kt

k;t1

 ln s



kt

s



k;t1

(11)

. Multiplying by wkt and summing over common goods, yields

(1  ) ln P



t

P



t1

= (1  ) k2
t1;twkt ln pkt

pk;t1

(1  ) k2
t1;twkt ln kt

k;t1

k2
t1;twkt ln s



kt

s



k;t1

(12)

!

P



t

P



t1

= k



pkt

pk;t1

wkt 2

4k2
t1;t 

s



kt

s



k;t1

! wkt

1

3

5 = k2
t1;t

0

@

pkt

pk;t1

 

s



kt

s



k;t1

! 1

1

1

A

wkt

(13a)

For the RW normalization, wkt =

1

Nt1;t

,



CCV

t =

P



t

P



t1

= k



pkt

pk;t1

 1

Nt1;t 2

4k2
t1;t 

s



kt

s



k;t1

! 1

Nt1;t(1) 3

5 (14)

which is their CCV index8.

However, an unweighted mean puts as much importance on a very widely

consumed good as one with a trivial share. Consider the normalization with the

SV weights instead, wkt = !



kt,

k2
t1;t

!



kt

kt  1 (15)

8 This is RW equation (9) without the tilde notation.

4
. Then (13a) becomes

P



t

P



t1

= k



pkt

pk;t1

!



kt

= SV

t

(16)

because

k2
t1;t 

s



kt

s



k;t1

! !

kt

(1)

 1 (17)

. Therefore, taste-shock bias from (5) is exactly zero



taste-shock bias

k2
t1;t!



kt ln kt

k;t1



 ln SV

t  ln CCV

t  0 (18)

.

If the weights were last periodís shares, wkt = s



k;t1

, the index would

be

P



t

P



t1

= Gt1;t

2

6

4

k2
t1;t 

s



kt

s



k;t1

!s



k;t1

1

3

7

5 (19)

where Gt1;t = k2
t1;t 

pkt

pk;t1

sk;t1

is the geometric means index with base

period (lag) shares.

Using current period shares, wkt = s



k;t,

P



t

P



t1

= k



pkt

pk;t1

s



k;t 2

6

4

k2
t1;t 

s



kt

s



k;t1

! s



k;t

(1) 3

7

5 (20)

. Using Tˆrnqvist weights, wkt =

1

2



s



kt + s



k;t1



,

P



t

P



t1

= T Qt1;t

2

6

6

4

k2
t1;t 

s



kt

s



k;t1

!

1

2 (s



kt+s



k;t1)

1

3

7

7

5

(21)

where T Qt1;t denotes the Tˆrnqvist price index. In each case, (5) di§ers.9

9Zadrozny (2020) uses generalized CES preferences instead,which allow for nonhomotheticity. However, the same normalization (9) is required, where the it parameters are

analogous to the kt RW parameters.

5
3 Comparison of Calculated Indexes with Alternate Preference Normalizations

CCV type indexes with alternative normalizations were constructed to compare to the CCV and SV indexes. The data used was chosen to be as similar as

possible to RW. However, RW use AC Nielsenís Homescan data, which is proprietary data of a household survey of purchases from grocery and drug stores

in the U.S. Homescan data for the same time span as RW, 2004-2014, was not

available for this study. Instead, I use BLSís Öve years of data of AC Nielsenís

Scantrak data, which contains the universe of transactions data of all large grocery and drug store chains in the U.S., so this is not meant to replicate the exact

results. The data contains weekly and monthly total quantities and unit value

prices for goods deÖned at the detail of UPC, which generally identiÖes an exact

good. Following RW, the data is aggregated to the quarterly level, and index

changes from 4th quarter to 4th quarter were calculated, from 4th quarter 2005

- 4th quarter 2009. The data is divided into categories called product groups,

which were aggregated using a Laspeyres formula with 4th quarter 2005 as the

base period. The elasticities of substitution were estimated separately for each

product group, using the methodology in Broda & Weinstein (2010), which in

turn uses the Feenstra (1994) method, to be as consistent as possible with RW.

Figure 110 and the table below reports the index changes for the CCV,

Fisher, Sato-Vartia share normalization (SV index), Fisher, Tˆrnqvist share

normalization, lagged share normalization, and current (lead) share normalization. The "taste change bias" is the di§erence between these indexes and the

SV index. As shown above, the normalization using SV weights is exactly the

SV index, with zero bias. The standard Fisher index is very close, and the Tˆrnqvist share index is slightly below. As reported in RW, the equal shares index

and lag shares index are greatly below the SV index, and except for the Örst

year, similar to each other, implying upward bias in a SV index. However, the

lead shares index is the opposite, being greatly higher and implying downward

bias in the SV index. These di§erences are very signiÖcant since they compound

over time. For example, the average di§erence between the lead share index and

the SV is 1.5%/year, which over ten years would compound to a di§erence in

index levels of 16.25%.

10 These results are analogous to RW Figure V, and the lagged share normalization is what

they refer to as the "CUPI - Initial Shares". RW include the Fisher index for comparison.

6
Figure 1: Index Relatives with Di§erent Preference Normalizations, 4th qtr to

4th qtr

Index Period

2006Q4 2007Q4 2008Q4 2009Q4

Sato-Vartia 1.0183 1.0447 1.06060 0.9851

Fisher 1.0184 1.0453 1.06186 0.9857

Lag Shares 1.0027 1.0241 1.03495 0.9532

Lead Shares 1.0367 1.0575 1.07676 1.0000

Tˆrnqvist Shares 1.0193 1.0401 1.05521 0.9758

Equal Shares (CCV) 1.0239 1.0285 1.02943 0.9559

Inverse Share 1.0278 0.9786 0.93114 0.8564

Lead Share Squared 1.0330 1.0628 1.07532 0.9964

Exponential Share 1.0239 1.0285 1.02945 0.9559

Additional normalizations are shown in Figure 2 and the table, including

weighting by the inverse share, squared lead share, or and exponential of the

share, for weights of wkt =

1

!

kt

k

1

!

kt

;

s

2

kt

ks

2

kt

; and exp(1+s



kt)

k exp(1+s



kt)

respectively. The

inverse and exponential share weighted indexes imply upward bias in the SV of

greatly di§erent magnitude, while the lead share squared index implies the

opposite.

More generally, a weighted mean always lies between the minimum and

maximum values, depending on the weights. The index is a weighted geometric

7
Figure 2: Additional Index Relatives with Di§erent Preference Normalizations,

4th qtr to 4th qtr

8
mean of

xkt =

pkt

pk;t1

 

s



kt

s



k;t1

! 1

1

(22)

. Therefore,

x

min

kt 

P



t

P



t1

 x

max

kt

. The highest xkt over good and time, x

max

kt , is about 4800%, while the lowest

xkt, x

min

kt , is about 86% deáation. Therefore, to yield the desired index I

choice

t

,

pick a weighting on the maximum xkt, w

max

kt such that

I

choice

t =



x

min

kt w

max

tk 

x

min

kt 1w

max

tk (23)

and place a trivial weight on all other xkt values,

wkt =

w

max

kt if k is max

1  w

max

kt if k is min

" otherwise

(24)

. Thus, if a statistical agency wanted to incorporate changing preferences by

this method, they could always claim they were doing it by choosing a set of

weights that yielded exactly the same indexes that they produced before!

A simple example shows how di§erent weights a§ect the index. "Taste-Shock

Bias" is only positive for the equal weighting normalization if product sales

shares diverge, as can be seen from (14) when the geometric mean of the shares

rises between periods. The problem with the equal weighting normalization

is that smaller shares have a proportionately higher relative change. Suppose

there are only two goods, with shares = .8 and .2, which then change to .9 and

.1, with  = 4. The second term in the r.h.s. of (13a) becomes

k2
t1;t 

s



kt

s



k;t1

! wkt

1

=



:9

:8

:5

3



:1

:2

:5

3

= (1: 019 8) (0:890 90) = 0:908 56 < 1

(25)

. Higher weighting on higher share goods could reverse this, such as wkt = s



kt,

so that

k2
t1;t 

s



kt

s



k;t1

! wkt

1

=



:9

:8

:9

3



:1

:2

:1

3

= (1: 036 0) (:977 16) = 1: 012 3 > 1

(26)

. The weight on the higher share goods therefore determines whether this term

moves the index upwards or downwards. Depending on how share ratios are

correlated with prices ratios (determined by the elasticity), this determines how

the index compares to a SV index.

9
4 Conclusions/Discussion

Since any index could be rationalized by the choice of a particular set of

weights, the decision to adjust for changing preferences by normalizing them

is empty. However, it is troubling that what is being measured, the cost of

a certain standard of living for certain preferences, may become irrelevant if

consumer preferences do change signiÖcantly. With additional data, it may be

possible to connect preferences between two periods by measuring the changes

in preferences, thus making a larger preference mapping that includes both

periods. But prices and quantities of goods are not su¢ cient.

5 References

1. Balk, Burt M., "Changing Consumer Preferences the Cost-of-Living Index:

Theory and Nonparametric Expressions", Journal of Economics 50 (1989):

157-169.

2. Feenstra, Robert C. and Marshall B. Reinsdorf, "Should Exact Index

Numbers Have Standard Errors? Theory and Application to Asian Growth",

Hard-to-Measure Goods and Services: Essays in Honor of Zvi Griliches

(1997). University of Chicago Press, pp. 483-513.

3. GabÛr-TÛth, Enikˆ, and Philip Vermeulen, "The relative importance of

taste shocks and price movements in the variation of cost-of-living: evidence from scanner data" (2020) Unpublished paper.

4. Martin, Robert, "Revisiting Taste Change in Cost-of-Living Measurement", BLS Economic Working paper # WP-515.

5. Muellbauer, John, "The Cost of Living and Taste and Quality Change",

Journal of Economic Theory 10 (1975): 269-283.

6. Pollak, Robert A. The Theory of the Cost-of-Living Index (1989). Oxford

University Press on Demand.

7. Redding, Steven, and David Weinstein, "Measuring Aggregate Price Indices with Taste Shocks: Theory and Evidence for CES Preferences", The

Quarterly Journal of Economics 135 (2020): 503-560.

8. Ueda, Kozo, Kota Watanabe, and Tsutomu Watanabe, "Product Turnover

and the Cost-of-Living Index: Quality versus Fashion E§ects", The American Economic Journal: Macroeconomics 11 (2019): 310-347.

10