James McAndrews: Review of the experience of fielding the survey of
consumer expectations
Remarks by Mr James McAndrews, Executive Vice President and Director of Research of
the Federal Reserve Bank of New York, at the Barclays Global Inflation Conference,
New York City, 24 June 2014.
* * *
Accompanying charts can be found at the end of the speech.
Good afternoon. It is a pleasure to be here today to talk with you about some of the research
being conducted by the New York Fed on the measurement of household expectations,
including households' expectations of inflation. The views expressed are my own and do not
necessarily reflect those of the Federal Reserve Bank of New York or of the Federal Reserve
System. To begin, I'd like to acknowledge some colleagues of mine who have managed our
household survey project and conducted much of the research that I'll speak about today
- namely Olivier Armantier, Giorgio Topa, Wilbert van der Klaauw, and Basit Zafar.
In 2012, my colleague Simon Potter spoke to this conference about the approach that
researchers at the New York Fed adopted in rolling out a household survey of expectations.
Today, I will report on the survey, which is the result of more than six years of research,
including a pilot survey that was conducted for about five years, and the published survey
that we ran for a year prior to its first publication in January 2014.
Starting in the fall of 2007, the New York Fed has conducted a long-term research project to
improve our understanding of existing measures of consumer expectations, and to explore
the feasibility of expanding and potentially improving these measures. We developed and
then tested new questions on expectations, designed measures of individual forecast
uncertainty, and collected information on expectations and decisions for a broader range of
household behavior than is currently gathered by existing surveys. In general, we wanted to
understand better the process of forming expectations and updating those expectations, as
well as the links between expectations and consumer behavior.
In pursuing this project, we collaborated with the RAND Corporation, other Federal Reserve
Banks, academic economists, and psychologists with expertise in survey design. We
conducted in-depth cognitive interviews and fielded psychometric survey modules utilizing
RAND's American Life Panel, and various convenience samples created by researchers at
Carnegie Mellon. Finally, we conducted many experimental surveys as part of the American
Life Panel internet survey. Incorporating the lessons we learned from these initiatives led to
the final design and implementation of our Survey of Consumer Expectations.
Our goal throughout has been to collect timely and high-quality information on consumer
expectations and decisions. We were particularly interested in filling in what we viewed as
gaps in existing sources of information on household behavior, including data on household
finance, labor, and housing market expectations. In addition, we decided that a rotating panel
structure, one that tracks the same individuals over time, was necessary for many research
and policy applications. Having a panel reduces the variability induced by changes in
composition month to month and increases the precision of any measured changes in
expectations.
Before we implemented the survey, our proposal was reviewed and approved by the Bank's
senior management. Next, we issued a request for proposals and received three bids from
independent survey organizations to field our survey design. In the end, we chose to work
with the Demand Institute, a partnership of the Conference Board and Nielsen.
The resulting Survey of Consumer Expectations is a nationally representative monthly
internet-based survey. Its rotating panel consists of about 1,200 household heads. We find
that the internet approach allows us a great deal of flexibility to ask new questions, and it
makes it considerably easier to pose probabilistic questions and to run experiments. It is also
- and this is an important consideration - the most cost-effective mode of collection; at the
same time, there is also some evidence of higher response accuracy to personally sensitive
questions. Our sample is based on the Conference Board's Consumer Confidence Survey
sample, which is drawn from U.S. postal addresses. Respondents are compensated with a
payment of $15 per survey, and surveys vary in length from fifteen to thirty minutes. Our
firsttime response rate is about 60 percent, and we find that respondents have somewhat
greater-than-average education and income. The average response rate among repeat
respondents is more than 80 percent. We use weights based on the American Community
Survey to construct summary statistics for our public releases.
The survey consists of a core monthly module on expectations about macroeconomic and
household-level variables and an additional quarterly module that allows us to address
special topics. Our core module contains questions about survey respondents' expectations
for inflation, wage growth, home prices, various commodities, household income and
spending, taxes, government debt, credit access, and job search. So far, quarterly modules
have included questions on housing and employment, work history, job search, retirement,
financial literacy, consumption and savings behavior, student loans, and various field
experiments. These quarterly modules are developed by Federal Reserve staff in
collaboration with academic consultants and the Nielsen team. First, however, new questions
are evaluated using cognitive interviews, and every new module is pilot-tested before being
put into the field.
The Bank began publishing the results of the Survey in January 2014, after having fielded
the Survey since December 2012. We introduced it through a series of posts on our Liberty
Street Economics blog; eventually, after a suitable embargo period, we plan to release to the
public all the micro data from the Survey.
Approach to and benefits of eliciting probabilistic beliefs
As I mentioned earlier, an important innovation of our Survey is that, in addition to asking the
respondent for a point forecast, it uses probabilistic questions to elicit a respondent's density
forecast. By doing so, we aim to collect a more accurate and complete representation of
individuals' subjective expectations, and the degree of uncertainty that they attach to their
forecasts.
Our method for eliciting density forecasts builds on a large body of empirical work, led by
economist Charles Manski, which involves the use of probabilistic survey question formats to
measure a respondent's beliefs about an uncertain future event. Our specific question format
was further tested during the experimental pilot phase of the project. Our density forecast
question asks respondents to assign probabilities to pre-determined intervals or bins for
future outcomes. For example, as shown on Slide 2, for year-ahead inflation, we ask for the
"percent chance" that inflation will be between 0 percent and 2 percent, 2 percent and
4 percent, 4 percent and 8 percent, etc., with similar bins used for deflation. In a similar
fashion, we solicit respondents' density forecasts for year-ahead earnings growth, house
price change expectations, and three-year ahead inflation.
For each individual respondent, the resulting density forecast enables us to construct
individual measures of central tendency (such as the density mean or median) and
uncertainty (such as the density interquartile range). To compute these, we use each
individual's responses to the probabilistic questions to parametrically estimate the underlying
forecast density function as proposed by Engelberg, Manski, and Williams in a 2009 Journal
of Business and Economic Statistics article. More specifically, when a respondent assigns a
positive probability to three or more bins, we assume an underlying generalized beta
distribution; and for fewer than three bins, we assume a triangular distribution. An example of
such a fit is shown on Slide 3.
Our findings over the past five years, based on different samples and different survey
platforms, indicate that individuals are as willing and able to respond to well-written
probabilistic questions as they are to traditional attitudinal questions on the same subject.
Moreover, they do so sensibly, with responses to probabilistic questions having internal
consistency and measurement reliability. The uncertainty expressed in consumers' density
forecasts is reliably related to other measures of uncertainty. For example, individuals who
express higher levels of uncertainty in their subjective probability distribution are more likely
to report a range when they are originally asked for their point forecast, and the width of this
self-reported range is positively correlated with measured uncertainty. In the case of inflation,
we find that uncertainty about future inflation is negatively related to self-assessed
responsibility for investment decisions, planning horizons for financial decisions, and the
respondent's performance on a financial literacy measure.
In addition to providing a measure of uncertainty, an important advantage of density
forecasts over point forecasts is that they remove ambiguity about which (if any) measure of
central tendency an individual's point forecast corresponds to. Measures of central tendency
derived from density forecasts are strongly correlated with point forecasts, and they have
very similar average values; however, for roughly half of the responses, the point forecast
falls in either the top or bottom quartile of the individual's forecast density, away from the
center. Thus, interpersonal comparisons of point forecasts reflect not just differences in
beliefs but also differences in what distribution statistic is being reported as point forecast by
different survey participants. Density forecasts therefore allow for a more accurate
measurement of disagreement among forecasters by using a common measure of central
tendency (for example, the mean or the median of individuals' subjective probability
distribution).
Findings on monthly expectations
The Survey of Consumer Expectations provides a broad range of measures that are used
routinely in our continuous assessment of the economy. We publish the monthly findings of
our survey on the second Monday of each month on the New York Fed's website.
The Survey page is organized into three sections: inflation, labor market, and household
finance. As shown in Chart 1, inflation expectations have been quite stable over the past
twelve months, at both the one-year and the three-year ahead horizons. Our preferred
measure is based on the median of the individual respondents' density means, but we also
report the median point forecast as well as the interquartile range of the density means
across respondents to highlight the dispersion of expectations across consumers. I will return
to this aspect later.
We also report home price change expectations and price change expectations for various
commodities. In our latest survey in May 2014, these expectations remained stable for the
most part. Interestingly, medical care price change expectations have been gradually
declining since December 2013, dropping from around 11 percent to 9.5 percent. The
expected cost of a college education has also been dropping since the start of 2014. These
results are shown in Charts 2 and 3.
For the labor market, we report the median expected earnings growth over the next twelve
months. This figure has been stable at around 2 percent over the sample period, with a small
increase to 2.4 percent over the winter that has since been reversed, as seen in Chart 4. We
also track workers' perceived chances of losing their current job, of leaving their current job
voluntarily, and of finding another job in three months should they lose their current one.
Quits, layoffs, and the job-finding rate are important flow measures, and workers'
expectations about these events enable us to gauge households' perceptions of future labor
market conditions. As I will mention later, these expectations seem to be meaningfully related
to job search behavior and other economic decisions by households.
Finally, the household finance section contains time series of expected household income
growth, expected household spending growth, changes in the amount of taxes that
households expect to pay, as well as perceived credit availability - both relative to one year
ago and looking ahead to one year from now. As Chart 5 indicates, household income
growth expectations declined slightly to 2.3 percent in May, but remained in the middle of the
narrow band (2.0 - 2.6) observed over the last twelve months. On the other hand,
perceptions of credit availability continued to improve slightly in May, as shown in Chart 6.
For most of our charts, the website gives additional demographic detail, breaking down each
time series by income, education, age, numeracy, and region of the country. Further, for our
main measures (inflation, home price changes, and earnings growth) we also report the time
series of forecast uncertainty. One interesting finding - consistent with our prior research - is
that forecast uncertainty about inflation is significantly higher for low education, low income,
low numeracy respondents. These findings can be seen in Chart 7.
Summary of past research
Let me now briefly describe some of the basic research we have carried out through our
consumer surveys. We have focused on two broad questions to date. First, we have
investigated the connection between inflation expectations and behavior in a financially
incentivized investment field experiment. In this study, survey respondents were asked for
their inflation expectations. Subsequently, within the course of the same survey, they were
asked to choose between two possible financial investments, one that gives a fixed nominal
return after twelve months, and another that yields a return indexed by inflation, again after
one year. Respondents were told that two survey participants would be drawn at random and
be paid according to their choice in the investment after twelve months.
Theory predicts that respondents would be more likely to pick the inflation-protected
investment if they expect higher inflation. Indeed, we find a strong association between
expectations and actual choices in the experiment. Furthermore, those who express higher
uncertainty about future inflation are also more likely to choose the inflation-protected
investment. Both patterns are in accordance with the theory. Interestingly, the connection
between expectations and experimental choices is weaker for respondents with low
education, numeracy, and financial literacy. Our research thus shows that consumer
expectations elicited through surveys are correlated in a meaningful way with actual
behavior. We also find that expectations inform behavior in other contexts: for instance,
workers who express a higher perceived chance of losing their current job over the next
twelve months also search harder for a new job and exhibit a drop in spending plans relative
to the present over the same time horizon.
The second area of research related to our consumer survey has focused on the formation
and updating of inflation expectations. Through another field experiment embedded in our
American Life Panel survey modules, we have examined how respondents update their
expectations after receiving information about either past food price inflation or future
inflation forecasts by professional forecasters. Before the information treatment, and again
after the provision of information, respondents were asked for their inflation expectations. We
also asked for respondents' priors about the information that was provided, to see whether
those who have larger perception gaps are more likely to update their expectations, and if
they would do so by a larger amount.
We found that respondents revise their expectations in the direction predicted by theory after
receiving new information. Further, the extent of revisions is correlated with the size of their
perception gaps. Finally, respondents are more receptive to the new information when the
uncertainty expressed in their baseline expectations is greater. All three findings are
consistent with rational, Bayesian updating of expectations.
Moving beyond average effects, we also found that our information treatment leads to a
significant decline in the cross-sectional dispersion of inflation expectations, and causes the
distribution of inflation expectations to converge toward its center - particularly for those who
had high baseline expectations and were less informed ex ante. In fact, average revised
expectations were nearer to actual realized CPI inflation as a result of our intervention. This
is an encouraging result: it suggests that policymakers could partially influence the
highexpectation right-tail of the inflation expectations distribution through public information
campaigns in the spirit of our information treatments.
Why heterogeneity for consumers, and why are expectations high?
An important and well-known distinctive feature of survey inflation expectations of individual
consumers and firms - when compared with that of professional forecasters - is the much
larger dispersion across individuals' beliefs about future inflation and the greater
rightskewedness, or long right-tail, of this distribution.
A number of different explanations have been proposed for the larger heterogeneity in
beliefs, the larger proportion of extreme high responses, and the subsequent higher mean
inflation forecast among consumers, summarized in Slide 11. First, in our research, we have
found a tendency among a subset of respondents to think about specific prices, and
especially those most salient to them, in coming up with a forecast. These often tend to be
prices of goods that changed the most or are most volatile, such as food and gas prices. As
a result, respondents who think about specific prices tend to report higher inflation
expectations. Respondents who instead think more about a broader overall measure of
inflation tend to report lower forecasts.
Several alternative explanations that have been proposed in the recent literature for the large
heterogeneity in beliefs focus on differences in information sets or in expectation-formation
processes. These include sticky information models, in which new information is slow to
diffuse through the population (Mankiw and Reis, 2002), perhaps because agents only
probabilistically pay attention to experts or to news (Carroll, 2003). They also include noisy
information models, in which agents form expectations based on noisy private signals
(Woodford, 2001). Other explanations are based on the use of some form of adaptive
learning (Evans and Honkapohja, 2001), switching between different prediction rules
(Branch, 2004); or learning from lifetime inflation experiences (Malmendier and Nagel, 2013;
Madeira and Zafar, 2012).
In our information-based experiment that I discussed earlier, we found that cross-sectional
disagreement (variance) in expectations falls after our information treatments, which is
consistent with a sticky-information model and points to the importance of private information
and information constraints. We also found evidence of gender differences in updating
behavior, which in turn may reflect differences in gathering and evaluating new information.
Numeracy, financial literacy, and education also appear to play a role in explaining
dispersion across respondents, as seen in Chart 8. In our research, we have found extreme
inflation expectations, especially high values, to be associated with lower financial literacy
and numeracy skills, and with being relatively under-informed about past inflation or
inflationrelated facts.
Finally, there exists suggestive evidence of respondents forecasting under asymmetric loss,
with respondents appearing to be averse to under-estimating inflation. Whereas it is common
to assume the mean is being reported - implying that respondents generate forecasts under
symmetric square loss - the evidence suggests that some individuals' forecasts are biased
away from the mean, suggesting forecasting using an asymmetric loss function.
Heterogeneity in loss aversion would contribute to the cross-sectional heterogeneity of
inflation expectations. Moreover, asymmetric loss would generate a mean bias that will vary
in size with the uncertainty of individual expectations, and hence is likely nonconstant over
time. This evidence points to another potential advantage of density forecasts: their reporting
may be less affected by loss aversion than simple point forecasts.
While there are several possible explanations for the large dispersion of inflation forecasts
among consumers, this heterogeneity is typically higher than that observed in various
surveys of professional forecasters. In closing, I would like to speculate on a few potential
reasons as to why the dispersion measured among professional forecasters may be too low
relative to a "true" dispersion in expectations.
First, asymmetric incentives may induce a desire to conform. If professional survey
respondents have beliefs that are far from the consensus, they might face an asymmetric
payoff: a small positive payoff for being correct, but a decidedly negative payoff for being
incorrect. This may cause a tendency to converge on a consensus belief. Similarly, in pricing
TIPS, a trader with a nonconformist view faces costs in taking that view, so even if ultimately
proven correct, the costs of establishing such a position may deter the nonconformist view's
from being reflected in market prices. Second, professional forecasters may be overly
exposed to similar "world views," sharing similar data, reports, and analyses. This may lead
to an information aggregation bias of sorts, in which forecasters share the same information
set, which is an aggregate of finer information, and which may result in the loss of some
important information.
The next charts describe the median four-quarter-ahead CPI forecast and the dispersion
around that median for the Blue Chip survey and the U.S. Survey of Professional
Forecasters. While there is significant overlap in the two sets of respondents, there is an
important difference between the two surveys: respondents are anonymous in the U.S.
Survey of Professional Forecasters, whereas they are identified by name in the Blue Chip
survey. The asymmetric payoffs described earlier may therefore have more of an impact in
the Blue Chip survey than in the U.S. Survey of Professional Forecasters, leading to less
dispersion in the former than in the latter.
Chart 9 shows that the median forecast is substantially the same across the two surveys.
However, as seen in Chart 10, there is some evidence of lower heterogeneity among
forecasters in the Blue Chip survey compared with the Survey of Professional Forecasters,
especially in recent periods, which, in retrospect, were periods of high fundamental
uncertainty in the economy. In both periods, during the run up to the financial crisis and its
aftermath, most forecasters were mistaken about future growth rates and inflation rates by
relatively large amounts. Both surveys of professional forecasters exhibit significantly less
dispersion than the Survey of Consumer Expectations - even when focusing only on
"sophisticated" respondents with high education and numeracy.
This suggests that there may be some scope for the sort of mechanisms I described, which
may induce the "fundamental" heterogeneity in beliefs to be suppressed in surveys of
professional forecasters. Consequently, household surveys may add important information
on "fundamental" heterogeneity of beliefs relative to professional surveys, as the latter may
reflect, in part, relativistic payoffs or shared information sets which produce an excessive
consensus of views.
Thank you for listening to me today; I would be happy to take some questions on our
research on inflation expectations.
BIS central bankers' speeches 7
8 BIS central bankers' speeches
BIS central bankers' speeches 9
10 BIS central bankers' speeches
BIS central bankers' speeches 11
References
Armantier, O., S. Nelson, G. Topa, W. van der Klaauw, and B. Zafar. 2012. "The Price is
Right: Updating of Inflation Expectations in a Randomized Price Information Experiment."
Federal Reserve Bank of New York Staff Reports, no. 543, revised January 2013.
Armantier, O., W. Bruine de Bruin, G. Topa, W. van der Klaauw, and B. Zafar. 2011.
"Inflation Expectations and Behavior: Do Survey Respondents Act on Their Beliefs?" Federal
Reserve Bank of New York Staff Reports, no. 509.
Branch, W. A. 2004. "The Theory of Rationally Heterogeneous Expectations: Evidence from
Survey Data on Inflation Expectations." The Economic Journal 114 (July): 592-621.
Bruine de Bruin, W., W. van der Klaauw, J. S. Downs, B. Fischhoff, G. Topa, and
O. Armantier. 2012. "The Effect of Question Wording on Consumers' Reported Inflation
Expectations." Journal of Economic Psychology 33, no. 4 (August): 749-57.
Bruine de Bruin, W., W. van der Klaauw, and G. Topa. 2011. "Expectations of Inflation: The
Biasing Effect of Thoughts about Specific Prices." Journal of Economic Psychology 32, no. 5
(October): 834-45.
Carroll, C. D. 2003. "Macroeconomic Expectations of Households and Professional
Forecasters." Quarterly Journal of Economics, 118, no. 1 (February): 269-298.
Engelberg, J., C. F. Manski, and J. Williams. 2009. "Comparing the Point Predictions and
Subjective Probability Distributions of Professional Forecasters", Journal of Business and
Economic Statistics 27, no. 1 (February): 30-41.
Evans, G. W., and S. Honkapohja. 2001. Learning and Expectations in Macroeconomics.
Princeton, New Jersey: Princeton University Press.
Madeira, C., and B. Zafar. 2012. "Heterogeneous Inflation Expectations, Learning, and
Market Outcomes." Federal Reserve Bank of New York Staff Reports, no. 536, revised June
2014.
Malmendier, U., and S. Nagel. 2013. "Learning from Inflation Experiences." University of
California, Berkeley, working paper.
Mankiw, N. G., and R. Reis. 2002. "Sticky Information versus Sticky Prices: A Proposal to
Replace the New Keynesian Phillips Curve," The Quarterly Journal of Economics, 117, no. 4,
(November): 1295-328.
Woodford, M. 2001. "Imperfect Common Knowledge and the Effects of Monetary Policy",
National Bureau of Economic Research working paper, no. w8673.