Svante Öberg: The Riksbank's forecasting performance
Speech by Mr Svante Öberg, Deputy Governor of the Sveriges Riksbank, at Stockholm
University, Stockholm, 26 November 2007.
* * *
Introduction
"When one remembers how foolishly one thought and how wrong one was and how different
things have usually been and how the world has been affected by strange metamorphoses,
one has less and less respect every year for forecasts." This is a fairly well-used quotation
from Alf Henriksson. And they are wise words. But at the Riksbank we have no choice, as
the interest rate affects inflation and the rest of the economy with some time lag. Our
decisions must therefore be based on forecasts. The question I wish to discuss today
concerns our forecasting performance.
Evaluation is important for independent central banks
Experiences show that the work on maintaining low and stable inflation is made easier if the
central bank can act independently. At the same time, greater independence makes
demands on the central bank to be open and to clearly explain its decisions. There are
several reasons why this is important.
One important aspect is that openness makes it easier to evaluate and to require
accountability, which is essential for a central bank's legitimacy. Openness enables the
general public to understand what the central bank is doing. This in turn makes demands on
the central bank to be clear in its analysis and message. While the openness makes it easier
to evaluate monetary policy, one can say that the evaluations are also an important part of
the openness.
Only a couple of decades ago central banks were fairly secretive. Very little of what was
carried out became known to the general public. There have been considerable changes
since then. The process of development from uncommunicativeness to increasing openness
that has taken place at central banks around the world has probably contributed to the
success of monetary policy. What I mean here is that they have succeeded in anchoring
inflation expectations at a low and stable level.
The Riksbank is one of the most open central banks of all. We began publishing our inflation
forecasts in 1997 and since then we have gradually become more open in our view of
economic developments. Other central banks are also moving towards greater openness.
For instance, only a few days ago the Federal Reserve in the United States began publishing
forecasts four times a year instead of twice. They have also extended the forecast horizon
from two to three years, just as we at the Riksbank did a couple of years ago.
It is thus essential that monetary policy is evaluated and discussed. This applies both to the
principles behind monetary policy and to how well we succeed in meeting our objective. To
make this possible there must be a clear basis for the decision. It must be clearly shown
which principles we have based our monetary policy decisions on and what information our
conclusions are based on.
As monetary policy affects the economy with some time lag, the monetary policy decisions
must be based on forecasts, as I mentioned at the start. Good bases for decisions in the
form of accurate forecasts are thus an important factor in the success of monetary policy. But
future developments in inflation and other variables are uncertain. For one thing, there is the
uncertainty linked to the statistical methods and models, as certain fundamental parameters
vary over time. For another thing, there are a number of other factors that can affect inflation
in a way that is very difficult to predict. For instance, increased geopolitical unease can push
up the oil prices, or an unexpected outbreak of a disease among animals can lead to a rapid
rise in food prices.
When central banks make forecasts, it is therefore increasingly common that they put a band
around the forecasts that measures their uncertainty. These bands help provide information
about the uncertainty in the forecasts and also make it easier to communicate the monetary
policy alternatives that may arise in the future.
But what do we know about the forecasting performance of the Riksbank?
Forecasting performance is evaluated both internally and externally
Forecasting performance can be evaluated in many different ways. There are a number of
statistical methods that can be used to analyse how accurate the forecasts are. One means
of measuring accuracy is to compare the forecasts with outcomes. The problem is that one
doesn't know whether a forecasting error is due to the forecasting method being inadequate,
or to something genuinely unpredictable. It is therefore common to compare accuracy with
forecasts from simple time series models to gain a perspective on whether the forecasting
errors are unusually large or small.
We have made a number of evaluations of our own forecasting performance and we have
also been evaluated by external experts. In recent years, we ourselves evaluated some of
the models used in the forecasting work. Various types of model are used in the forecasting
work to produce the best possible base for an assessment of economic developments. A
study that will soon be published has investigated the forecasting precision of our general
equilibrium model and the Bayesian VAR model.1 Another study has evaluated our indicator
models.2
In addition we publish information to enable an assessment of the monetary policy conducted
over the past 2-3 years in a special appendix to the first Monetary Policy Report of every
year. It contains, for instance, comparisons of our forecasts with other forecasts and with
outcomes. The material we produce in our annual evaluations forms an important basis for
the Riksdag Committee on Finance's report on monetary policy.
Every year the Committee evaluates at its own initiative the monetary policy conducted
during the previous years. They have done so since 1999 when the Riksbank became
independent. Last year we also had two external researchers, Francesco Giavazzi and
Frederic Mishkin, who made a more in-depth evaluation of monetary policy. This was
commissioned by the Committee on Finance and covered the entire ten-year period of
inflation targeting.
New study of the Riksbank's forecasting performance
As a step in the evaluation work, three economists at the Monetary Policy Department of the
Riksbank recently made a comprehensive analysis of our forecasting performance during the
period 2000-2006.3 They have analysed the forecasts of the most important variables for
monetary policy - GDP, employment, productivity and inflation.4
In the study they have tried to estimate how accurate the forecasts have been on average by
comparing them with outcomes. To gain a perspective of the accuracy, our forecasts have
also been compared with forecasts from a simple time series model. The so-called AR
models, which are often used as a standard comparison in evaluations of this type, quite
simply function so that one adjusts the variable in question with the aid of its own historical
data. The type of time series model chosen for the comparison is, despite its simplicity,
difficult to surpass, according to several international studies.
Moreover, they have analysed how far ahead we can make forecasts with good accuracy,
that is to say what forecasting memory the different variables have.5 They have also looked
at whether there are tendencies to systematically over-estimate or under-estimate the
different variables, that is, whether the forecasts have any bias.
What does the analysis show about our forecasting performance?
Let us first look at the entire period and compare our forecasts for GDP, employment and
CPIX with outcomes. CPIX is a measure of underlying inflation, which until recently was
known as UND1X. Hardly surprisingly, it appears that the forecasts have varied in accuracy
(Figure 1). For instance, we have under-estimated employment growth recently (Figure 2).
Nor did we predict the sharp rise in inflation spring 2001 in our assessments the previous
year. However, we were more successful in capturing the downswing in inflation during 2003
(Figure 3). But how do things look in more detail for the different variables?
Forecasts for GDP growth
With regard to the GDP forecasts, we have periodically been accurate and close to the
outcome, while we have fallen short on other occasions (Figure 1). It is often particularly
difficult to make forecasts in an economic turnaround.
Compared with the time series model that is used to gain a perspective of the forecasting
memory, we make better GDP forecasts up to a year ahead. With regard to the forecasting
memory for GDP, that is, how far ahead we can make a forecast with some information
value, this stretches four quarters ahead (Figure 5). This agrees with experiences in other
countries. One result of an evaluation that analysed the forecasting memory of Canadian and
US macro variables shows that it is often limited to the first forecasting year.6
Another means of analysing the forecasting performance is to look at the correlation between
forecast and outcome (Figure 9). The correlation shows the strength in the relationship
between forecasts and outcomes. A correlation of one year is a perfect relationship, if it is
zero there is no relationship at all, while minus one is a perfect negative relationship. The
correlations shall thus preferably be positive and as close to one as possible.
One year ahead the correlation between the GDP forecast and outcome is just over 0.6 and
two years ahead it is around 0.1. This supports the picture we obtained when we compared
out forecasting memory with the time series models. We were better than the model up to
one year ahead and we had a forecasting memory four quarters ahead.
Forecasts for employment
If we instead look at employment forecasts, we believed that employment would rise more in
2003 and 2004 than it actually did (Figure 2). Instead, productivity growth was repeatedly
stronger than expected. But on the other hand, employment has been stronger than
expected in our assessments over the past year. The forecasts we have made for
employment growth are on the whole more accurate than those of the time series model and
the forecasting memory stretches eight quarters ahead (Figure 6).
If, on the other hand, we only look at developments over the past two years, the Riksbank
has succeeded in forecasting employment about as well as the time series model. This
means that we have not succeeded in adding any further information on top of what the
simple model can capture.
How is the relationship between employment forecasts and outcome? The correlation is
almost 0.9 one year ahead and just over 0.4 two years ahead (see Figure 9). The correlation
is stronger than for GDP and agrees with the picture we obtained when we compared the
forecasts with the time series model. The employment forecasts appear on average to have
been more accurate than the GDP forecasts.
Forecasts for productivity
Productivity is a central variable in the monetary policy analysis. The most common measure
of productivity is labour productivity, that is, the ratio between GDP and the number of hours
worked. It has not been possible to evaluate this measure here, as there is not enough
historical forecasting data on the number of hours worked. Instead the ratio between GDP
and employment has been used as an approximation.
The analysis shows that our forecasts have under-estimated productivity growth on all
forecasting horizons (Figure 4).7 The time series model is on the whole slightly more
accurate than our forecasts have been during this period. The forecasting memory for
productivity has only extended one quarter ahead (Figure 7). The same applies to the
correlation between forecasts and outcomes. It is 0.7 one quarter ahead, while it is zero both
one and two years ahead (see Figure 9).
Inflation forecasts
When it comes to inflation measured as the annual rate of increase in the CPIX, we have
made fairly accurate forecasts during a large part of the period evaluated, that is, 2000-2006
(Figure 3). Something we missed was the sharp upturn in inflation in spring 2001. This was
primarily due to an outbreak of mad cow disease which led to rising food prices. But since
then we have made inflation forecasts that have been close to the outcomes. For instance,
the downswing in inflation in 2003 was captured to a greater extent than the upswing at the
beginning of 2001. The inflation forecasts are on the whole more accurate than the time
series models. The forecasting memory for the CPIX extends eight quarters ahead (Figure
8).
The correlation between forecasts and outcome is almost 0.8 one year ahead and just over
0.5 two years ahead (see Figure 9). None of the other variables has as high a correlation two
years ahead. The fact that the correlation wanes in time is natural as the uncertainty in the
forecasts increases.
Is there any bias one or two years ahead in the forecasts?
Let us now look at whether there is any bias, that is, systematic over-estimation or
underestimation in the forecasts one and two years ahead. The smaller the bias a forecast has, the
better it is.
One year ahead the bias for annual GDP growth is very small. However, two years ahead
the forecasts have been almost been four tenths too high on average, compared with the
outcomes, which is not negligible (Figure 10).
With regard to the annual development in employment, the bias is just under three tenths
one year ahead and two tenths two years ahead. Employment has been under-estimated in
both cases, that is, the outcomes have on average been higher than in the forecasts.
Productivity has also been under-estimated both one and two years ahead. Productivity has
a bias of around one tenth one year ahead and just under seven tenths two years ahead. We
have thus made a relatively large under-estimation of productivity growth two years ahead.8
With regard to the annual change in inflation measured as the CPIX, the forecasts have on
average been almost three tenths lower than the outcomes one year ahead. However, the
forecasts two years ahead have on average been two tenths higher than the outcomes,
which can be regarded as a minor deviation.
What conclusions can we draw from this? The bias is relatively small for most of the
variables. There are no clear signs of any systematic errors in the forecasts, with the
possible exception of productivity. The rate of increase in productivity has been
systematically under-estimated during the period. However, it should be added that none of
the variables has any statistically significant bias.
Comparison with other forecasters
The study has also compared the Riksbank's forecasting errors with the National Institute of
Economic Research (NIER) and Consensus Forecasts panel, a compilation of forecasts from
a number of different forecasters. It is clear from the study that the Riksbank's forecasts for
GDP growth have been among the best compared with other participants in the Consensus
panel (Figure 11). On around half of the occasions we have made more accurate forecasts
than 80 per cent of the other forecasters. The inflation forecasts measured as the annual rate
of change in the CPI have in almost 75 per cent of cases been more accurate than half of the
forecasters in Consensus panel (Figure 12). In a comparison of quarterly GDP and CPIX
forecasts, it is shown that the Riksbank has on the whole made slightly more accurate
forecasts than the NIER. At the same time, it should be said that the differences in
forecasting performance are not statistically significant.
If we compare the results with an earlier forecast evaluation made by the Riksbank, there are
some similarities. In this evaluation we found, for instance, that the NIER has made better
forecasts of productivity than we have, at the same time as we have made better forecasts of
CPIX inflation.9
There are of course a number of difficulties in making correct comparisons of this nature,
which is also indicated in the study. It is important to note that the Riksbank made forecasts
under the assumption of an unchanged repo rate up to the publication of the second Inflation
Report of 2005. With this method CPI inflation should actually be over-estimated in the
shortterm if the interest rate is cut. It is actually only with effect from this year, when we have
begun to make forecasts based on our own interest rate path, that the forecasts fully mirror
our views of the future.
Another point is that the forecasts are rarely made at the same point in time, which can of
course make accuracy more difficult. However, the analysis here is designed to ensure this
problem is as small as possible.
Forecasts are uncertain and we mark this by an uncertainty band
Forecasts are uncertain and one is not always right. Most of those who make forecasts are
very well aware of this. Back in 1997 when we began publishing forecasts of inflation we
reported uncertainty bands around the forecasts. As uncertainty increases over time, the
uncertainty band takes the form of a fan (Figure 13). We now also publish uncertainty bands
for the GDP and repo rate forecasts.
The uncertainty that is reflected in the bands can be divided into three parts. It reflects the
fact that we are uncertain about the current situation, regarding how the economy functions,
and about what shocks might occur during the forecast period. The uncertainty bands are
constructed so they are symmetrical around our main scenario and for GDP growth and the
inflation rate the breadth of the bands is calculated with the aid of our earlier forecasting
errors. This gives an incentive to make good forecasts, as large forecasting errors bear the
penalty of broad uncertainty bands and the risk of reduced credibility in the forecasts.
The uncertainty band for the repo rate cannot be calculated in the same way, as we have
only begun to make our own forecasts for the repo rate this year. There are quite simply
insufficient historical forecasts to use as a base. The uncertainty band for the repo rate has
instead been constructed with the aid of forecasting errors among the so-called implied
forward rates (Figure 14). The implied forward rates reflect market expectations of the repo
rate. One problem with this method is that the implied forward rates have systematically
over-estimated the future repo rate path for a number of years. However, the over-estimation
has been removed from the forecasting errors, with reference to the fact that the forward
rates also consist of a risk premium that should not be included when using them for
forecasts.
Different central banks have chosen different methods of illustrating the uncertainty in the
forecasts and we regularly compare notes with one another.
We are continuing to develop our forecasts
The conclusion from this review is that the Riksbank's forecasts on the whole are rather
good, particularly one year ahead.
We have made good forecasts for all variables, with the exception of productivity, one year
ahead. This is reflected in the fact that we have made more accurate forecasts than the time
series model and also in the fact that the correlations are relatively high between forecasts
and outcomes. In addition, the bias is small for all variables one year ahead.
We have a forecasting memory two years ahead for both employment and inflation. Although
the correlations between forecasts and outcomes are weak one year ahead, they
nevertheless show a clear relationship. On the other hand, the study implies poor forecasting
memory for GDP and productivity two years ahead. The correlations between the forecasts
and outcomes are also weak for these variables two years ahead. We have seen that the
bias is larger for most variables two years ahead. GDP and inflation have been
overestimated slightly during the period studied, and employment and productivity have been
under-estimated.
Our inflation forecasts, particularly those for CPIX, have good qualities such as high
correlation with outcomes and long forecasting memories. It is worth pointing out that this is
the variable that is most important for monetary policy. At the same time, the uncertainty
band shows that probability that inflation will be within the Riksbank's tolerance interval of 1-3
per cent two years ahead is only around 75 per cent, despite our forecast being that it will be
two per cent.
On the other hand, our forecasts for the real economy are not as accurate as the inflation
forecasts, although they compare fairly well with the time series model's forecasts.
Our forecasts also compare fairly well with those of other forecasters. The result of the
comparison with other forecasters shows that our GDP forecasts up to one year ahead are
often among the most accurate. This also applies to the CPI, but here the difference is
smaller compared with the NIER and the Consensus panel.
This does not mean that we have not occasionally made considerable forecasting errors. But
the errors are not so systematic that there is reason to believe they are due to inadequate
forecasting methods. And nor does this mean that we cannot or do not wish to become even
better. We are constantly working to improve our methods. In some areas we need to work
extra hard to improve our forecasting performance. One such area is productivity.
It is important that we continue to regularly assess monetary policy, and particularly with
regard to our forecasts, both in our annual assessments and in other ways. This is necessary
both to be able to gain better knowledge of which variables are difficult to forecast and to see
where it is necessary to develop methods and analysis. A major advantage of publishing our
forecasts is that external experts can also evaluate us.
Thank you for inviting me here today!
Figure 1. GDP outcomes and forecasts
Annual percentage change
6 6
5 5
4 4
3 3
2 2
1 1
Figure 2. Employment outcomes and
forecasts
Annual percentage change
4,0 4,0
3,0 3,0
2,0 2,0
1,0 1,0
0,0 0,0
-1,0 -1,0
mar-01 mar-02 mar-03 mar-04 mar-05 mar-06 mar-07
Sources: Statistics Sweden and the Riksbank
Figure 3. CPIX outcomes and forecasts
Annual percentage change
3,5 3,5
3,0 3,0
2,5 2,5
2,0 2,0
1,5 1,5
1,0 1,0
0,5 0,5
0,0 0,0
-0,5 -0,5
mar-00 mar-01 mar-02 mar-03 mar-04 mar-05 mar-06 mar-07
Sources: Statistics Sweden and the Riksbank
Figure 4. Productivity outcomes and
forecasts
Annual percentage change
5,0 5,0
4,0 4,0
3,0 3,0
2,0 2,0
1,0 1,0
0,0 0,0
-1,0 -1,0
-2,0 -2,0
mar-01 mar-02 mar-03 mar-04 mar-05 mar-06 mar-07
Sources: Statistics Sweden and the Riksbank
Figure 5. Forecasting memory, GDP
Percentage points
0,5
Riksbank
Time series model (AR(1))
Standard deviation
0,4
0,3
0,2
0,1
0,0
1 2 3 4 5 6 7 8
Kvartal
Sources: Statistics Sweden and the Riksbank
Figure 6. Forecasting memory, employment
Percentage points
0,5
0,4
0,3
0,2
Riksbank
Time series model (AR(1))
Standard deviation
0,1
0,0
1 2 3 4 5 6 7 8
Kvartal
Sources: Statistics Sweden and the Riksbank
Figure 7. Forecasting memory, productivity
Percentage points
0,7
Riksbank
Time series model (AR(1))
Standard deviation
0,6
0,5
0,4
0,3
0,2
0,1
0,0
1 2 3 4 5 6 7 8
Kvartal
Sources: Statistics Sweden and the Riksbank
Figure 8. Forecasting memory for the KPIX
Percentage points
0,7
0,6
0,5
0,4
0,3
0,2
Riksbank
Time series model (AR(1) with seasonal dummies)
0,1
Standard deviation
0,0
1 2 3 4 5 6 7 8
Kvartal
Sources: Statistics Sweden and the Riksbank
Figure 9. Correlations between forecasts
and outcomes for the different variables
Annual rates of change
1,0
1,0
0,8
0,8
0,6
0,6
0,4
0,4
0,2
0,2
cent
0,0
0,0
Per
1 2 3 4 5 6 7 8
-0,2
-0,2
GDP
-0,4
-0,4
Employed
CPIX -0,6
-0,6
CPI
-0,8
-0,8
GDP per employed
-1,0
-1,0
Forecast horizon (quarterly)
Sources: Statistics Sweden and the Riksbank
Figure 10. Mean error for forecasts, an
annual rate
Percentage points
0,80 0,80
0,60 0,60
0,40 0,40
0,20 0,20
0,00 0,00
-0,20
-0,20
-0,40 -0,40
-0,60 -0,60
-0,80 -0,80
0 1 2 3 4 5 6 7 8
GDP Employment CPIX CPI GDP/ empolyment
Forecast horizon (quarterly)
Sources: Statistics Sweden and the Riksbank
Figure 11. Ranking for GDP following year
Per cent
Note. The Y-axis shows how large a percentage of times analysts have been better than the percentage of
analysts as illustrated in the X-axis. In order to see how large a proportion of times the Riksbank has been
better than 50 per cent of other analysts, see 50 on the X-axis, then study the red column on the Y-axis.
100
Riksbank
NIER
Consensus
Time series model (AR(1))
80
60
40
20
0
50 60 70 80 90
Sources: Statistics Sweden, the National Institute of Economic Research, Consensus Forecasts and the
Riksbank
BIS Review 138/2007
Figure 12. Ranking for the CPI following year
per cent
Note. The Y-axis shows how large a percentage of times analysts have been better than the percentage of
analysts as illustrated in the X-axis. In order to see how large a proportion of times the Riksbank has been
better than 50 per cent of other analysts, see 50 on the X-axis, then study the red column on the Y-axis.
100
Riksbank
NIER
Consensus
Time series model (AR(1))
80
60
40
20
0
50 60 70 80 90
Sources: Statistics Sweden, the National Institute of Economic Research, Consensus Forecasts and the Riksbank
Figure 13. The CPIX with uncertainty band
Annual percentage change
4 4
90%
75%
50%
UND1X
3 3
2 2
1 1
0 0
-1 -1
dec-02 dec-03 dec-04 dec-05 dec-06 dec-07 dec-08 dec-09 dec-10
Sources: Statistics Sweden and the Riksbank
BIS Review 138/2007
Figure 14. Repo rate with uncertainty band
cent, quarterly averages
Per
7 7
90%
75%
50%
6 6
Reporänta
5 5
4 4
3 3
2 2
1 1
0 0
dec-02 dec-03 dec-04 dec-05 dec-06 dec-07 dec-08 dec-09 dec-10
Source: The Riksbank