Daniel Mminele: Forecasts, errors and what we can learn from them
Keynote address by Mr Daniel Mminele, Deputy Governor of the South African Reserve
Bank, at the Reuters Economist of the Year Award, Johannesburg, 5 August 2011.
* * *
1. Introduction
Good morning ladies and gentlemen.
Thank you to Thomson Reuters for inviting me to address you this morning and to celebrate
with you your achievements in economic forecasting. May I take this opportunity to extend
my sincere congratulations to the winner of this prestigious award to be handed out this
morning, and also to congratulate all of the nominees present here today.
I was very tempted to start off my remarks this morning by telling a little joke about
economists, but decided against it, as I have been warned that an economist is someone
who did not have enough personality to become an accountant.
I promise to keep my remarks this morning quite brief.
2. Economic forecasting tested during the crisis
Thomas Kida1 noted, "...the amount of knowledge we have in a certain area will not help us
predict what will happen if the events are inherently unpredictable."
The recent global financial crisis and recession has proved to be a humbling experience for
economists, financial market participants, policy makers, regulators and many others that
derive their income from economic forecasting, understanding the world economy and that
which governs it. The crisis was not foreseen by most forecasters, although there was a
minority who had been warning of a pending disaster for some time. The failure to forecast
the "Great Recession" caused the Queen of England herself to ask why it had not been
foreseen. The turmoil of recent years has prompted critics to maintain that many economic
models, and particularly macroeconomic models based on rational expectations, have been
proven to be fundamentally flawed.
Much of the criticism directed towards forecasting and macroeconomic analysts has probably
been misplaced. It has never been so much about economic models being wrong, as it is
about understanding that all models are limited - by definition - and that models that limit
certain types of relationships will eventually fail when those relationships in the real world
change.
In other words, the right criticism is to say that economic models constructed in particular
ways need to be used and understood in similarly particular ways. A model that does not
include household balance sheets and what drives them will not tell you if you will have a
household debt crisis. By the same token, a model that says that future wage inflation will be
a perfect function of expected future inflation will always be at odds with reality. Criticising
economists for thinking that their models replace what actually happens in the real world may
be satisfying, but it is also off-key.
By the same token, placing complete faith in an economic forecast or model is equally
unhelpful, as we know that no model can fully capture the complex relationships we find in
the real world. Therefore models should never be used mechanistically. Forecast errors
cannot tell us anything useful about financial and economic panic and crashes unless the
models on which they are based account for the build up of imbalances that become
unsustainable. Clearly, the conditions for sustainability also need to be included. For
example, the ability of Greece to manage its current debt load differs depending on how fast
the Greek economy can grow.
The mistake was to allow so much of the global community, especially the financial markets
and policy makers, to rely on economic analysis that placed little emphasis on structural
factors and balance sheets and which ignored the herd behaviour occurring in financial
markets. It was not a model of real estate prices that failed to see the absurdity of ever-rising
prices in the US housing market, but the assessment of the bull market in property by many,
often famous, economists.
So how might we think about dealing with this problem of economic models and forecasting
that will be by definition insufficient and, in fact, wrong, and the need for robust and useful
economic analysis both of what models tell us and what they don't?
Certainly part of the answer lies in making progress with developing models that are useful in
the appropriate contexts. For the Reserve Bank, a major step forward in modelling came with
the implementation of the inflation targeting monetary policy framework in 2000. More
accurate modelling and forecasting activities were (and are) required for inflation targeting,
and the Research Department of the Bank set up a so-called "suite of models". This ranges
from single equation, high frequency data models to longer-term structural multivariate
models, and also varies according to the combination of economic theory and statistical
techniques applied.2
It is useful to supplement the forecasts from structural models with information obtained from
higher frequency models to establish the starting point for the forecast trajectory. This
approach allows the forecasters to optimise the positive characteristics of a particular model
type and use the different approaches to improve the trajectory of the forecast and minimise
errors.
The suite of models approach can be useful to obtain different forecasts for the same
variable which can then be pooled together to obtain the final forecast. Models differ in their
focus and consequently in their characteristics and transmission mechanisms. Different
outcomes are useful in gaining new perspectives on what is driving particular outcomes.
There are of course limits to this approach. Too many different model properties make the
assessment of the transmission mechanisms ambiguous and complicates the interpretation
of the results.
The success of any modelling and forecasting endeavour should be measured not by the
sophistication or complexity of the techniques used or the accuracy of the data employed,
but rather the degree of forecast error minimisation and the quality of the analysis that it
supports.
More and better information is critical to model building, forecasting and analysis. And here,
the Reuters Survey provides a valuable service to the modelling activities of the Reserve
Bank, and of course other forecasters. It is useful to compare forecasts to shed light on how
other participants view the future and how they quantify the most important transmission
channels of the economy. The analysis of surveys over time, moreover, tells us something
about the nature of shocks hitting the economy, how the economy adjusts to the shocks, and
the extent of policy reaction expected by the markets.
In our recent discussions of the economy, considerable attention was devoted to
understanding the combination of inflation, growth, and monetary policy reaction functions
exhibited in private sector forecasts. Greater uncertainty in the global economic environment
and newer data from international and domestic sources made it critically important to gather
more views of conditions in the world and understand how they were being interpreted by the
markets. A levelling-off of commodity prices and weaker economic growth in many
economies presented significant differences from how we understood the world even two
months ago.
There have been other changes too, including on the domestic front, such as signs of weaker
second quarter growth rates. Our task is to try to peer through these various changes, in part
to ask ourselves hard questions about our forecasts and test them against potential
scenarios.
Scenarios, in turn, imply probabilities of events actually transpiring, and here our analysis
has had to cross the boundaries into other fields. We noted for instance in the Monetary
Policy Committee Statement our concerns about the debt crises in advanced economies and
considered in our discussions the possibilities of how this crisis might be resolved.
This general increase in uncertainty has become a characteristic of the post-recession world,
and one that appears especially hard to cast off. Uncertainty demands a greater application
of judgement across a broader range of perspectives than in times when variables move in
clear directions and risks are low.
3. Conclusion
These thoughts prompt us to recognise that flexibility is a key ingredient of useful
macroeconomic forecasting exercises, even as flexibility may seem to violate the very idea of
a model. This tension is in fact quite important. The inflexibility of a model's fixed
relationships help us to assess whether today's actual outcomes will continue in the future or
adjust in line with the historical relationships reflected in the model. As real world outcomes
change, we in turn ask ourselves what is wrong with our model relationships or what other
conditions have changed to generate outcomes beyond what the model suggests.
In short, the framework used to produce forecasts must be dynamic and able to change
should circumstances require. One area for future work for us in the Reserve Bank is to
incorporate the lessons of the financial crisis and recession into how we use our macro
econometric and other models.
The financial crisis that has occurred was so unexpected that very few, if any,
macroeconometric models captured the inter-linkages between the real and the financial spheres.
Beyond interest rates, macro econometric models generally do not include financial
transmission mechanisms. Clearly this needs to change, but we also need to recognise that
even much more extensive coverage will not compensate for intensive regulatory focus on
financial and asset markets and the balance sheets of institutions operating in them.
And from a policy-making perspective, we need to recognise that whatever progress we
make in refining and calibrating our models, and achieving a higher level of their
sophistication, they will never be able to absolve us from our responsibility to ultimately
exercise judgement in reaching any policy decisions.
Thank you.