Simon M Potter: The advantages of probabilistic survey questions
Keynote address by Mr Simon M Potter, Executive Vice President of the Markets Group of
the Federal Reserve Bank of New York, at the IT Forum and RCEA Bayesian Workshop,
Rimini, Italy, 19 May 2016.
* * *
Accompanying slides can be found on the Bank of Finland's website: Slides (PDF).
I would like to thank Marco Del Negro, Giorgio Topa, and Wilbert van der Klaauw for their excellent assistance in
the preparation of these remarks, Luis Armona, Daniele Caratelli and Joseph Fiorica for able research assistance,
and colleagues in the Federal Reserve System for their insightful comments and suggestions.
Good afternoon. It's a pleasure to talk with you about a topic that intrigues me as a former
academic researcher, and that's of significant importance to me in my current role as head of
the New York Fed Markets Group and as manager of the System Open Market Account for
the Federal Open Market Committee (FOMC): the measurement of policy-relevant
expectations through surveys.
It is entirely fitting to talk about this topic here in Italy, given the Bank of Italy's long history of
research and experimentation on expectations data. This history includes extensive work on
data gathered from surveys using a probabilistic question format - the focus of my talk today.
Since 1989, the Bank of Italy's Survey of Household Income and Wealth has used this
question format to ask households about their expected labor or pension earnings in the next
year. The findings from these surveys have provided new insights into consumer behavior,
such as the impact of subjective earnings uncertainty on precautionary saving.1 The Bank of
Italy was also a pioneer in using probabilistic questioning to elicit expectations from
businesses in its 1993 Italian Survey of Investment in Manufacturing. Data from this survey
have been used to analyze the impact of uncertainty about future product demand on the
investment decisions of Italian manufacturing firms.2
As always, the views expressed are my own and do not necessarily reflect those of the
Federal Reserve Bank of New York or of the Federal Reserve System.
Why are expectations data useful?
The importance of compiling high-quality data on the expectations held by economic agents
has been increasingly recognized in both academic research and policymaking. Most
economic decisions involve uncertainty, and are therefore determined not only by
preferences but also by expectations for future outcomes. Typically, neither preferences nor
expectations are directly observed - a condition that poses a significant challenge to
understanding economic behavior. Although different combinations of preferences and
expectations may be consistent with the same observed behavior, they could result in
different responses to shocks or to alternative policies and could therefore have different
implications for public policy.3 The standard way to address this problem within economic
models has been to assume rational expectations - that is, expectations that are consistent
with equilibrium behavior in the model. In nearly all cases considered to date, this leads to
the conclusion that expectations of economic agents are identical or will converge over time.
While this approach has given us invaluable insights in a multitude of models and
applications, it faces a major empirical problem: in practice, economic agents hold widely
differing views about the future path of economic variables, as I will illustrate throughout
these remarks. Thus, recent research has focused on alternative assumptions regarding the
nature of expectations held by economic agents. Measuring subjective expectations directly
can inform our modeling assumptions on the information sets available to agents, and on the
nature of the expectations formation and updating process.4
There are many Bayesians in the audience today. However, for the benefit of those who are
not as familiar with Bayesian terminology, let me quickly define what I mean by "subjective
expectations" or beliefs. This term refers to the personal probability distributions that
individuals hold over uncertain events. Apart from the laws of probability and rules about how
beliefs are updated with the arrival of information, there are no restrictions on these
probability distributions, hence the use of the word "subjective."5 Further, individuals might
act according to their subjective beliefs without being able to fully articulate in standard forms
their underlying probability distribution. Thus, a need arises to elicit these views in order to
understand behavior more fully.
In monetary policymaking, central bankers have long pointed out the importance of
measuring the expectations of financial market participants, households, and firms -
especially with regard to inflation and the central bank's so-called "reaction function" to
changes in the economic outlook. For example, measuring market participants' expectations
for the policy rate path sheds light on both the central bank's effectiveness in communicating
its policy intentions and the transmission of anticipated future monetary policy actions into
financial conditions. Inflation expectations of households, firms, and other economic agents
influence a variety of decisions, including those related to consumption, saving, investment,
and the setting of prices and wages, as well as the determination of the real interest rates
that influence these decisions. The aggregation of these choices in turn determines realized
inflation in equilibrium. Thus, inflation expectations are a crucial determinant of actual
inflation and, as such, need to be carefully monitored. Further, the effectiveness of monetary
policy and central bank communication relies on longer-run inflation expectations being
wellanchored, making their reliable measurement very important for policymakers. More
generally, expectations by consumers and businesses about a number of household,
firmlevel, and economy-wide outcomes are increasingly useful inputs into a variety of forecasting
and "now-casting" models.
In addition to tracking measures of expectations implied by asset prices, central banks today
rely increasingly on survey-based measures of subjective expectations. At the New York
Fed, we conduct a diverse set of surveys to measure policy-relevant expectations. Today I
will talk about the surveys of households and market participants we conduct: the Survey of
Consumer Expectations (SCE), the Survey of Primary Dealers (SPD), and the Survey of
Market Participants (SMP). These surveys highlight the innovative and cutting-edge research
conducted at the New York Fed and represent the culmination of the Bank's decade-long
investment in the collection of better policy-relevant expectations data. As I will now explain,
much of this investment has been directed toward introducing probabilistic survey questions,
but we have also learned other important lessons about question design and survey mode,
particularly regarding the measurement of inflation expectations which I will also discuss. In
some cases such as results of the SPD and SMP the underlying probabilistic nature of
survey questions is immediately apparent, in the case of the SCE as I will describe the
underlying probabilistic question is not so apparent for many of the results.
Why use probabilistic survey questions?
A key feature of the New York Fed surveys is their focus on collecting rich quantitative data
on subjective expectations. Extending a practice with a longer tradition in the field of
psychology and in surveys of professional forecasters, economists, and other financial
experts, our surveys rely heavily on a probabilistic question format to elicit the likelihood
respondents assign to different future events. This approach builds on a large and growing
body of research, led by econometrician Charles Manski, that has demonstrated survey
respondents' willingness and ability to answer questions expressed in this way.6
Traditionally, surveys ask respondents to express their answers in one of a few ways: for
example, using a Likert scale (specifying whether an event is "very likely," "likely," or "not too
likely"), selecting the outcome they consider most likely ("Do you expect the unemployment
rate to be higher or lower than today?") or giving a point response ("By what percent do you
think prices will increase over the next year?").7 A problem with the Likert scale is that it does
not allow comparisons across respondents, because different respondents may interpret the
scales differently. A well-known illustration of the problem was provided by Daniel
McFadden, who looked at a health status measure that used a Likert scale ranging from
"excellent" to "poor." While 62 percent of Danish men reported their health to be excellent,
only 14 percent of French men did so, despite their enjoying two more years of life
expectancy. A problem with questions asking for the outcome considered most likely - that
is, questions asking for the modal outcome - is the ambiguous information content of the
responses. For example, in the case of an event with a binary outcome (where the event
either occurs or not), the modal outcome can be assigned a probability that ranges from
50 percent to 100 percent. Finally, a problem with questions asking for a point forecast -
"what do you think will be" or "by how much do you expect" - is that it is not clear whether the
estimate individuals provide represents a mean, mode, median, or something else. Research
also suggests that some respondents provide a point estimate that reflects varying degrees
of loss aversion rather than just a pure forecast.
Among the advantages of questions asking directly for the likelihood or "percent chance" of
different outcomes are ease of interpretation, comparability across respondents, and the
ability to measure subjective uncertainty. The consistency of a respondent's answers can be
checked using the laws of probability, and importantly, probabilistic questions permit
elicitation of the quantitative measures of belief and uncertainty typically required in
estimating economic models.
Researchers can use a variety of methods to check the validity and information content of
probabilistic subjective expectations data. First, one can look at response rates, reported
ease and clarity of the questions used to elicit expectations, the use of focal points in
responses, and the internal consistency of responses. Second, one can consider whether
reported expectations vary with observable characteristics in a predictable way.8 Third,
whenever possible and assuming a stationary environment, one can check for relevance of
reported expectations, looking at whether they are meaningfully related to future or past
realizations. Fourth, one can ask whether subjective expectations help predict actual
behavior. Finally, one can look at whether agents update their reported expectations in
sensible ways upon receipt of relevant information.
Based on these criteria, a literature that has grown rapidly since the mid-1990s has
demonstrated the validity and information content of responses to probabilistic questions
about a broad spectrum of economic concepts. This evidence extends beyond the United
States and Europe to developing countries.9 Thus, probabilistic questions, with appropriate
instructions and visual aid tools, have been successfully used not only in the Survey of
Professional Forecasters and the U.S. Health and Retirement Survey (and its sister surveys
in Europe and Japan) but also in surveys of fisherman and farmers in India, aspiring migrants
in Tonga, poor households in rural Colombia, and junior high school students in Mexico.10
Some of these questions ask respondents to assign probabilities to various ranges of
outcome - so-called density forecasts.11 Generally, people across a wide range of countries
and walks of life appear to be able to answer probabilistic questions, and their expectations
are found to be useful in forming predictions of future economic outcomes.
The survey of consumer expectations
Turning now to the development and use of expectations surveys at the New York Fed, let
me first discuss our Survey of Consumer Expectations (SCE). Over the last ten years, we
extensively tested and eventually implemented a new survey of households, with the goal of
collecting timely and accurate information at high frequency on U.S. consumers' expectations
and decisions on a broad variety of topics. The SCE, launched in June 2013, was designed
to fill gaps in existing data collections about consumer expectations and outcomes, to
provide a more integrated data approach, and to take advantage of state-of-the-art survey
techniques. The SCE is implemented as a monthly, nationally representative survey of about
1,300 household heads.12 Recently the Bank of Canada implemented its own version of the
SCE, fielded at a quarterly frequency.
The SCE has various components, which are outlined on Slide 2. First, respondents are
asked a core monthly module of questions concerning their expectations about various
macroeconomic and household-level variables. These questions cover inflation expectations
and expectations regarding changes in home prices and the prices of various specific goods
and services, such as gasoline, food, rent, medical care, and a college education. The core
survey also asks for expectations about unemployment, interest rates, the stock market,
credit availability, taxes, and government debt. In addition, respondents are asked to report
their expectations about several labor market outcomes, including changes in earnings, the
perceived probability of losing their current job or leaving it voluntarily, and the perceived
probability of finding a new job. Moreover, respondents are asked about the expected
change in their household's overall income and spending. As I will describe in more detail
below, these expectations questions are fielded at various time horizons and with various
formats, including both point forecasts as well as density forecasts based on probabilistic
questions. The second component of the SCE contains a supplementary "ad hoc" module
each month on special topics.13 Finally, SCE respondents also fill out longer surveys each
9
See Delavande, Giné, and McKenzie (2011a) for a comprehensive review.
10
See Mahajan et al. (2008), Giné and Klonner (2007), McKenzie, Gibson, and Stillman (2013), Attanasio,
Meghir, and Vera-Hernández (2005), Attanasio and Kaufmann (2009), and Delavande, Lee, and Yoong
(2012).
11
More precisely, respondents are asked to assign probabilities to a set of mutually exclusive and exhaustive
ranges of continuous outcomes.
12
The SCE survey instrument is fielded by the Demand Institute, a nonprofit organization jointly operated by the
Conference Board and Nielsen.
13
Three such modules are repeated every four months, leaving three "floating" supplements per year on topics
that are determined as the need arises. The three repeating supplements cover credit access, job search and
retirement, and spending. Topics addressed so far in the "floating" supplement include the Affordable Care
Act, savings from lower gas prices, student loans, family leave, and use of insurance products. Together, the
core monthly module and the monthly supplement take about fifteen minutes.
quarter on various topics. These are up to thirty minutes in length and are separate from the
monthly survey.14
Before giving more details about the SCE questions and some background on the research
underpinning them, I would like to briefly point out two other important design elements of the
SCE. First, responses are collected using an online survey tool. While it has become harder
to obtain nationally representative samples with telephone and mail surveys, for which
response rates have been falling, the opposite has been true for online surveys. Moreover,
some evidence suggests that respondents are more likely to answer financial questions
posed in online surveys. In a recent randomized survey experiment conducted by colleagues
at the New York Fed and the Dutch central bank,15 the share of respondents reporting their
inflation expectations in online interviews slightly exceeded the share reporting their
expectations in face-to-face interviews.
In addition to being more cost effective, online surveys, when fielded at high frequency,
provide important flexibility by allowing questions to be added or changed at short notice. It is
thus possible to collect information on new economic and financial developments, such as
the impact of recent changes in the price of oil. Other attractive features of online surveys are
the ease of incorporating graphics and other visual refinements and the ease of conducting
randomized interactive experiments to analyze the updating of expectations and the links
between expectations and behavior.
A second key design element of the SCE is that it uses a rotating-panel sample design, with
approximately 1/12 of respondents rotating in each month, after which they stay in the panel
for up to 12 months and then rotate out. In addition to keeping the sample representative
over time, this rotational structure allows us to track the changes in individuals' responses
over time. In other words, we can compute changes in expectations over time after
differencing out individual idiosyncratic effects. This design contrasts with repeated
crosssectional surveys in which an entirely new sample is drawn each month. The panel structure
of our survey, with a more stable sample composition, therefore reduces volatility in
summary statistics due to a changing sample and increases the signal-to-noise ratio. Finally,
it is possible to use the panel to link short-term expectations to actual realizations, and
measure individual-level forecast accuracy and its implications in real time.
Designing survey questions for household inflation expectations
The launch of the SCE followed an extensive testing phase, dubbed the Household Inflation
Expectations Project (HIEP), which was initiated in 2006 to explore the feasibility of
implementing a new survey of consumer expectations with a focus on inflation
expectations.16
The HIEP had several objectives: to improve existing measures of consumer
expectations, to clarify the process of expectations formation and updating, and to study the
links between reported expectations and actual behavior in a variety of realms. The HIEP set
up a working group - composed of New York Fed and Federal Reserve System economists,
academic economists, behavioral psychologists, and survey design experts - that devised,
conducted, and analyzed a series of cognitive interviews and experimental surveys to
explore various dimensions of the planned new survey.
In particular, the HIEP analyzed the information content of the inflation expectations
questions in the University of Michigan Survey of Consumers, the primary source of
information on household inflation expectations at the time; tested alternative wording of
potential inflation expectations questions; studied the feasibility of eliciting individual
uncertainty about future outcomes for inflation, house prices, and earnings growth; and
introduced a panel dimension to the experimental data collection effort in order to study the
persistence of inflation expectations and their responsiveness to inflation surprises.
The HIEP considered different potential time horizons for its inflation expectations questions.
For the short-term expectations, we settled on the one-year-ahead horizon - similar to the
Michigan survey. We also examined the feasibility of asking for inflation expectations over a
longer time horizon. The Michigan survey asks respondents by what percent per year they
"expect prices to go up or down on the average, during the next 5 to 10 years".17 We
replicated this question in our experimental surveys, and found that the Michigan question
elicits a mixture of interpretations, with some respondents using a 10-year horizon and others
thinking about a 5-year horizon.18
From the results of our cognitive interviews and experimental surveys, we decided in the
SCE to elicit medium-term inflation expectations at the one-year two-year-forward horizon.
For example, in this month's edition of the SCE we ask respondents what they expect the
rate of inflation to be "over the 12-month period between May 2018 and May 2019." Our
testing suggests that respondents understand this format better than that of the 5-10 year
Michigan question; consumers are better able to provide their expectations over a specific
time period in the future, and there is little value in asking them about a longer time horizon.
In addition, we believe that the one-year two-year-forward horizon adopted in the SCE is
better suited to measure inflation expectations at the medium-term horizon that matters most
for central bankers, since monetary policy is expected to exert its full effect within that time
frame.19
With regard to question wording, it is important to note that the Michigan Survey asks about
changes in "prices in general." The HIEP tested three alternative wordings of a potential
inflation expectations question: we presented respondents with questions about the change
in "prices in general" (the wording used in the Michigan survey), the change in "prices you
pay," and the "rate of inflation." Compared with the "rate of inflation," the phrases "prices in
general" and especially "prices you pay" induced respondents to think more about their
personal price experiences and to focus on specific price changes for individual items, such
17
The Michigan survey's precise question format is as follows: First, respondents receive the question "What
about the outlook for prices over the next 5 to 10 years? Do you think prices in general will be higher, about
the same, or lower, 5 to 10 years from now?" Those who respond "stay the same" are then asked whether
they mean that prices will go up at the same rate as now, or that prices in general will not go up during the
next 5 to 10 years. Those who indicate that they mean prices will go up at the same rate are then given the
same follow-up questions as those who answer that they believe prices will be higher 5 to 10 years from now.
Respondents who answer that they expect prices to be higher [lower] 5 to 10 years from now receive the
question "By about what percent per year do you expect prices to go up [down] on the average, during the
next 5 to 10 years?" Only respondents who give a response over 5 percent are then asked the clarifying
follow-up question "Would that be [x] percent per year, or is that the total for prices over the next 5 to
10 years?" Respondents who answer "total" are then asked for a "per year" amount.
18
Further, the clarifying question used in the Michigan survey that asks whether respondents meant their
response to reflect price changes per year or over the entire time period induced significant revisions. The
follow-up question is administered only to respondents who give expectations over 5 percent, thus failing to
correct misinterpretations among those who gave lower responses. In our experimental surveys, we instead
administered the follow-up to everyone. We found that if we administered the follow-up question only to those
giving responses over 5 percent, then the median long-term expectation would be 4.3 percent instead of
3.7 percent in our test sample. Thus, we have reasons to believe that the 5 percent follow-up question in the
Michigan Survey leads to a systematic overestimation of actual average responses.
19
In contrast, for the Survey of Primary Dealers discussed later, we added a question in 2007 that directly
elicited uncertainty over CPI inflation from 5 to 10 years ahead. See Potter (2011, 2012).
as food or gasoline.20 In related research, we have shown that respondents tend to express
more extreme inflation expectations when they are prompted to think of specific prices in
answering the question rather than overall inflation.21 In the SCE, we therefore ask directly
for expectations about the rate of inflation or deflation.22
Probabilistic questions on the SCE
As I noted earlier, a key feature of our survey is its use of probabilistic questions. Following
the current practice in the literature for the elicitation of expectations about continuous
variables, we tested questions that elicited density forecasts by asking respondents to assign
the percent chance that the value of interest would fall within different pre-specified ranges or
"bins." For instance, for inflation, we ask for the subjective probability distribution over a
range of possible future inflation outcomes. This is shown in Slide 3.23
For each respondent, we fit a flexible parametric probability density function (PDF) to their
bin probabilities. We can then compute several moments of interest from the fitted density
forecast for each individual respondent, including measures of central tendency such as the
density mean or median, measures of uncertainty, such as the interquartile range, or IQR,
and measures of skewness.24
As I already noted, there are several important advantages to eliciting a density rather than a
simple point forecast. In a world characterized by pervasive uncertainty, density forecasts
provide a comprehensive representation of respondents' views about possible future
outcomes for the variables of interest. Importantly, density forecasts enable us to compute
the degree of uncertainty associated with each respondent's estimate. The measurement of
forecast uncertainty is important for policymakers in order to assess the extent to which
inflation expectations remain well-anchored. In addition, by eliciting density forecasts, we can
focus on a specific measure of central tendency and use that for cross-respondent
comparison and aggregation purposes. Research indicates that by forcing respondents to
consider the likelihood of different outcome ranges, density-based expectations measures,
20
In analyzing individual changes in expectations (from one month to the next) expressed by our SCE
respondents, we find that changes in inflation expectations are little correlated with changes in expectations
about future gasoline price changes.
21
Bruine de Bruin et al. (2011a) carried out two studies to see whether individuals who think about specific price
changes in forming their inflation expectations report more extreme and dispersed expectations because they
focus on more extreme specific price changes. In the first study, the researchers show that those who are
asked to report any or the largest individual price change tend to recall more extreme ones than those who are
asked to report the average price change, and subsequently report more extreme inflation expectations. In the
second study, the researchers show that among those who are asked about inflation expectations without first
being asked about individual price changes, about half nonetheless think about individual items. Those who
do so then report more extreme and dispersed inflation expectations.
22
In our open-ended cognitive interviews and experimental surveys, we found that respondents generally are
familiar with the term "inflation" and have a good understanding of the concept of inflation (Bruine de Bruin et
al. 2011a, 2012). Other studies have found that members of the general public are familiar with the term
"inflation" and have a basic understanding of what it means (Leiser and Drori 2005; Svenson and Nilsson
1986; Williamson and Wearing 1996).
23
An advantage of eliciting density forecasts in this way - as opposed, for example, to eliciting values of the
cumulative distribution function - is that answers are less likely to violate the laws of probability, such as
monotonicity of the distribution function. A simple instruction and visual tool showing a running total as
probabilities are assigned to bins leads to a negligible number of cases in which probabilities do not add up to
100 percent.
24
Following the approach described in Engelberg, Manski, and Williams (2009), we fit a generalized beta
distribution to the bin responses of each individual respondent whenever the respondent assigns positive
probability to three or more bins. When respondents only assign positive probability to one or two bins, we fit a
uniform or a triangular distribution, respectively.
such as the mean, are less prone to outliers than are one-off answers to point expectations,
resulting in smaller cross-sectional dispersion.25
Eliciting probabilistic beliefs and density forecasts presents a number of challenges: much
research is still needed on the best way to elicit such beliefs - for example, with regard to the
format of the questions; the number, width, and location of bins; whether to tailor bins to
individual respondents; and how to estimate the underlying continuous densities. The
elicitation of such beliefs involves many survey design choices, and the extent to which the
findings depend on the way in which the information is collected is not yet fully known.
In line with earlier findings in the literature, our research shows that survey respondents are
able and willing to provide density forecasts.26 In our experimental surveys, fielded as part of
RAND's American Life Panel, we observe negligible non- response rates to the bin
probability questions. Further, when respondents are asked point forecast questions, some
elect to provide a range, in order to express a degree of uncertainty in their forecast. We find
that the use of ranges in point forecast questions, as well as the widths of reported ranges,
are positively correlated with the number of bins to which respondents assign positive
probability in density forecast questions, and with measures of uncertainty based on the fitted
density estimates. Respondents also rate the density forecast questions only slightly harder
to answer than the corresponding point forecast ones. Finally, respondents' estimated
forecast uncertainty is meaningfully associated with month-to-month revisions in inflation
expectations: greater uncertainty is significantly correlated with larger revisions - qualitatively
consistent with Bayesian updating.
What is the best way to aggregate the information we elicit regarding probabilistic beliefs and
density forecasts? For binary outcomes, such as the perceived percent chance of losing
one's job over the next three months, we report the mean probability across respondents.
For density forecasts, we can compute several aggregate measures. Slide 4 shows a
stylized depiction of how we approach this. As our aggregate measure of expected inflation,
we report the median of individual density means. We pick the density mean, which
represents an individual's expectation for the rate of inflation, as our measure of central
tendency, and use the median to aggregate across respondents, since the median is less
sensitive to outliers. We can also look at medians of various percentiles of the individual
distributions.27
We measure aggregate forecast uncertainty as the median across respondents of the
individual IQRs. We also monitor the probability that respondents attach to extreme inflation
outcomes by computing, for instance, the median probability of deflation: in other words, the
median of the individual subjective probabilities assigned to outcomes with inflation less than
zero. Finally, to characterize the variation in inflation expectations across respondents, we
compute the dispersion of expected values: the IQR of individual density means. Again, we
can look at other percentiles of the distribution across respondents to detect possible
clustering and polarization.28 We publish at a monthly frequency many of the aggregate
measures of inflation expectations, as well as measures for a broad range of labor market
and financial expectations, on our survey website, shown on Slide 5. In addition to providing
national figures, we show trends for different demographic subgroups.
Current trends and additional research
So far, we have discussed the elicitation of density forecasts for a single variable of interest.
In addressing various policy questions, however, we may be interested in the subjective joint
probability distributions for two or more variables. For instance, if we are interested in
households' expectations about real wage growth, we may want to elicit the joint distribution
over future outcomes for both inflation and nominal wage growth. Measuring the joint
distribution of expectations over income and spending growth may also be very useful.
This is an area at the frontier of current research,