Exposition, climax, denouement : life-cycle evaluation of the recent financial crisis in the EU by linking the ESRB financial crisis database to the European Commission’s macroeconomic imbalance procedure scoreboard.
Date: 2019-10-27
Institutions: European Systemic Risk Board
Subjects: European Commission, database, economic disparity, financial risk, macroeconomics, monetary crisis

Working Paper Series 

No 102 / September 2019 

Exposition, climax, denouement 

Lif e-cycle evaluation  of  the recent f inancial crisis in the 
EU by linking the ESRB f inancial crisis database to the 
European  Commission’s Macroeconomic Imbalance 
Procedure Scoreboard 

By 
Szilard Erhart 

Abstract

The paper investigates the life-cycle of the 2008-2009 ﬁnancial crisis by linking the Macroe-

conomic Imbalance Procedure (MIP) Scoreboard of the European Commission to the crisis

database of the European Systemic Risk Board (ESRB). The novelty of the analysis is that

early warning capacity of MIP indicators is empirically tested in case of various crisis events

case by case (i) Currency/Balance-of-Payment/Capital ﬂow events, (ii) Sovereign crisis events,

(iii) Banking crisis events and (iv) Signiﬁcant asset price corrections in EU Member States.

Furthermore, we contribute to the literature by studying the predicting power of the MIP

Scoreboard in the identiﬁcation of the overheating in the economy in advance of crises (pre-

ventive arm of the MIP). We found that the predictive power of the MIP Scoreboard may

be twice as high to capture sovereign and Currency/Balance-of-Payment/Capital ﬂow type of

crisis events than its power to capture a banking crisis or serious asset price corrections. We

conﬁrm the results of earlier empirical studies that some MIP indicators perform relatively

well (current account and net international position) in all speciﬁcations. A simple composite

indicator based on the threshold breaches of MIP Scoreboard Indicators, performed in most

cases as good as the best individual indicator, and hence could be considered as an input to

a simple, rule based and accountable decision making.

JEL classiﬁcation: C40, G01, E44, E61, G28

Keywords: early warning system, macroeconomic imbalance procedure, signal approach,

financial crisis, boom and bust, systemic risk

1

1

Introduction

The paper reconsiders the role of the Macroeconomic Imbalance Procedure (MIP) Scoreboard and

its 11 headline indicators (see further details in Section 2.1). The European Commission introduced

the MIP in 2011 as part of the “Six-Pack” legislation, so as to react to the Global Financial Crisis

and to treat macroeconomic imbalances in the European Union (EU). The MIP Scoreboard aims at

identifying imbalances both ex-ante, during the early phase in the crisis life cycle (so as to trigger

pre-emptive actions) and also at monitoring imbalances ex-post (so as to trigger corrective actions).

In this paper, the predictive power of the MIP Scoreboard Indicators are tested for all three phases

of crisis life-cycle. Previous empirical studies on the MIP Scoreboard investigated only whether its

indicators could have helped predict the 2nd and 3rd PHASE of crisis life-cycles, e.g. certain types

of crisis events and economic downturns following crisis events (Figure 1) in EU Member States.

This paper also tests whether MIP indicators could help to signal the 1st PHASE of crisis events,

the overheating of the economy (preventive arm of the MIP).

Furthermore, we investigate systemically how the deﬁnition of a crisis event inﬂuences the pre-

dictive power of the MIP Scoreboard by linking the MIP Scoreboard to the European Systemic

Risk Board’s crisis database. We found that the MIP could have been able to signal two types of

crisis events with the highest probability: (i) Sovereign crisis events and (ii) Currency/Balance-

of-Payment/Capital ﬂow crises in EU Members States, but not banking crises. This result can

also help to reconcile the varying ﬁndings of earlier studies that deﬁned crisis events diﬀerently.

Csortos and Szalai (2014) used the cyclical GDP gap similarly to Domonkos et al. (2017), Boysen

et al. (2015) analysed the union of ﬁnancial crisis events and Knedlik (2014) the debt crisis events.

Our empirical ﬁndings conﬁrmed the hypothesis of Erhart et al. (2018) who argued that the real

house price growth indicator of the MIP and its current one-sided threshold in the MIP Scoreboard

is able to identify the overheating of the economy well, but not economic downturns.

We combine the signals approach of empirical studies and the composite indicator methodology to

evaluate the performance of the MIP Scoreboard. Such method has been proposed by the empiri-

cal study by Christensen and Li (2014) to capture the fragility of the economy prior to a ﬁnancial

stress event most eﬃciently. Lo Duca et al. (2017) also argued that the newly established ﬁnancial

crisis database conﬁrms that multivariate methods can improve upon univariate signalling models.

2

Figure 1: Life cycle of serious imbalances in EU Member States (Number of Member States in a
situation of overheating, crisis, downturn)

30

20

10

0

2,004 2,006 2,008 2,010 2,012 2,014 2,016

PHASE 1: Overheating
PHASE 2: Crisis events
PHASE 3: Downturn

*All 3 crisis phases are deﬁned in Section 2 and 3

Empirical works have concluded so far that the predictive power of the MIP Scoreboard Indica-

tors is low (Boysen et al. (2015); Knedlik (2014) and Csortos and Szalai (2014)). Furthermore,

the diﬃculty of early warning in case of macroeconomic surveillance is conﬁrmed by the fact that

diﬀerent studies identiﬁed diﬀerent indicators useful as alarm bells (Table 1). Csortos and Szalai

(2014) showed that only in the cases of the current account deﬁcit and the unemployment rate

were the prediction ratios better than the ratios of false alarms to alarms total. Boysen et al.

(2015) found that house prices, private sector debt, and private sector credit ﬂow are the best

early warning indicators of crisis events. Knedlik (2014) showed that the usefulness is the high-

est for the current account, net international investment position and nominal unit labour costs.

Domonkos et al. (2017) found that in the short run private sector debt is the best performing in-

dicator among the headline indicators, complemented by current account balances in the long term.

Knedlik (2014) also remarked that some indicators perform diﬀerently as early warning indicators

in diﬀerent countries. For instance, ULC performs excellently in Central and Eastern European

countries but poorly in the Eurozone. Public sector debt is a very important indicator for the

Eurozone but not for the rest of the European Union; the unemployment rate and export market

share are relevant early warning indicators in the Eurozone but not in Central and Eastern Euro-

pean countries.

Finally, diﬀerent explanations could be given as to why establishing eﬀective early warning systems

is a diﬃcult analytical and policy challenge. First, vulnerabilities are time and place dependent.

3

Second, early warning indicators may fail to trigger action and in many cases they did fail in

practice in the past, because policy makers were resistant to act on vague warnings in good times.

The MIP is deﬁnitely subject to these challenges (Erhart et al. (2018)).

Table 1: MIP empirical literature review : the best indicators to signal crisis events

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n
i

l
a
n
o
i
t
a
n
r
e
t
n
i

t
e
N

e
c
n
a
l
a
b

t
n
u
o
c
c
a

t
n
e
r
r
u
C
X

X
X X

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

l
a
r
e
n
e
G

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

l
a
i
c
n
a
n
ﬁ

l
a
t
o
T

e
t
a
r

t
n
e
m
y
o
l
p
m
e
n
U
X

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

l
a
e
R

s
e
r
a
h
s

t
e
k
r
a
m

t
r
o
p
x
E

d
e
t
a
ﬂ
e
d

,
x
e
d
n
i

e
c
i
r
p

e
s
u
o
H

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

e
t
a
v
i
r
P

t
b
e
d

r
o
t
c
e
s

e
t
a
v
i
r
P

X X X
X

t
s
o
c

r
u
o
b
a
l

t
i
n
u

l
a
n
i
m
o
N

X

Studies (in chronological order) Target variable
Csortos-Szalai (2013, 2014)
Boysen-Hogrefe et al. (2015)
Domonkos et al.(2017)
Knedlik (2014)

Downturn (cycl. GDP gap)
Crisis (UNION of syst., banking, sov. debt, currency)
Downturn (cycl. GDP gap)
Debt crisis

The rest of the paper is structured as follows. Section 2 describes our two data-sets (i) the MIP

Scoreboard and (ii) the ESRB’s Financial Crisis database. Section 3 presents the methodology.

Section 4 discusses the results on the signaling power of the MIP Scoreboard indicators in terms

of diﬀerent phases of the recent ﬁnancial crisis life-cycle. The conclusions are presented in Section

5.

4

2 Data

The analysis in this paper was based on two datasets: (1) the annual MIP Scoreboard dataset of

the European Commission and (2) the ESRB ﬁnancial crisis database.

2.1 The MIP Scoreboard Indicators

The MIP Scoreboard is a set of headline indicators aiming at the assessment of macroeconomic

risks (Commission (2012)). It consists of 14 indicators currently. Initially, the MIP Scoreboard was

built upon 11 indicators grouped into (I) ﬁve ’external imbalance indicators’, and (II) six ’internal

imbalance indicators’, which were augmented by additional 3 employment indicators Commission

(2015). However, these new headline indicators do not play a direct role in the identiﬁcation of

macro-ﬁnancial risks and do not trigger by themselves steps in the MIP, hence this report does not

cover them.

Headline MIP Indicators and their indicative upper and/or lower thresholds in parenthesis:

I. EXTERNAL IMBALANCE INDICATORS

- Current account balance: in % of GDP, 3 years average (upper: +6% and lower - 4%)

- Net international investment position: in % of GDP (-35%)

- Real eﬀective exchange rate: 42 trade partners, 3 years % change (-/+5)% for euro-area

countries and (- /+) 11% for non-euro-area countries)

- Export market shares: 5 years % change (- 6%)

- Nominal unit labour cost index: 3 years % change (+9% for euro-area countries and +12%

for non-euro-area countries)

II. INTERNAL IMBALANCE INDICATORS

- House price index, deﬂated: 1 year % change (6%)

- Private sector credit ﬂow: consolidated, in % of GDP (15%)

- Private sector debt: consolidated, in % of GDP (133%)

- General government sector debt: in % of GDP (60%)

- Unemployment rate: 3 years average (10%)

- Total ﬁnancial sector liabilities: non-consolidated - 1 year % change (16.5%)

In the MIP dataset, there are 336 observations for each indicators for the period between 2005 and

2016 (except for real house prices for which we had 333 observations) summing to 3693 observa-

5

tions (11 Indicators X 28 Member States X 12 years, 3 missing values). The great advantage of

the dataset is that it combines observations from the pre- and post-crisis (boom and bust) periods.

Hence, the database and the statistical analysis is less subject to possible biases stemming from

economic cyclicality.

2.2 The ESRB Financial Crisis database

The ESRB ﬁnancial crisis database provides the latest chronological summary of crisis periods to

support the calibration of models in macroprudential analysis. Lo Duca et al. (2017) discusses

how the ESRB database identiﬁes ﬁnancial crises by combining a quantitative approach based on

a ﬁnancial stress index with expert judgement from national and European authorities. The crises

database is updated on a regular basis. Alessi and Detken (2018) showed that there is a large

overlap between the widely used two datasets (Laeven and Valencia (2018) and ESRB) and the

results across the two are robust).

CRISIS TYPES IN THE ESRB DATABASE

1. Currency/Balance-of-Payment/Capital ﬂow

2. Sovereign

3. Banking

4. Signiﬁcant asset price correction

The ESRB ﬁnancial crisis database is a set of binary variables for all crisis types (1 if crisis,

and 0 otherwise) for the 28 EU Member States, in the period between 2005 and 2016 summing to

1344 observations (4 crisis types X 28 Member States X 12 years ).

3 Methodology

3.1 Signals Approach

We followed the best practice of numerous empirical studies on the MIP and used the so-called

’Signals Approach’, (Boysen et al. (2015), Knedlik (2014), Domonkos et al. (2017), Csortos and

Szalai (2014)). The idea of the signals approach is that an indicator or a system of indicators can

forecast a crisis, whenever it exceeds or falls below a certain threshold in a given forecast horizon

(see Boysen et al. (2015)). Such identiﬁcation of the best performing early warning system (EWS)

indicators could also be advantageous, because it could help design an optimal scoreboard or com-

6

Figure 2: Number of EU Member State crisis events by ESRB crisis types

20

15

10

5

0

2,004 2,006 2,008 2,010 2,012 2,014 2,016

Currency / BoP / Capital ﬂow
Sovereign
Banking
Signiﬁcant Asset Price Correction

posite measure.

In the framework of the signals approach an indicator signals a crisis, if it exceeds or falls be-

low a certain threshold. Overall, there are four cases (A, B, C, D) depending on whether a crisis

signal or the absence of a crisis signal was correct or incorrect.

A the number years, in which the indicator CORRECTLY provides signal.

B the number of years, in which the indicator INCORRECTLY provides signal.

C the number of years, in which the indicator INCORRECTLY provides NO signal.

D the number of years, in which the indicator CORRECTLY provides NO signal.

Table 2: Probability space of the Signals Approach

Signal

YES
NO

Crisis
YES NO
B
D

A
C

Signal performance evaluation

The less binding the thresholds, the more crises are correctly predicted (A) but at the same

time the number of incorrect signals (B) and, thus, the number of ‘false alarms’, eg. Type II

errors increases. In the same vein, a stricter threshold increases the number of cases, in which one

7

correctly predicts that no crisis occurs (D) but it also increases the number of crises without any

preceding signal (C) and, thus the number of Type I errors. The Commission aimed at striking a

healthy balance when setting MIP thresholds at prudent levels, in order to avoid both excessive

numbers of ‘false alarms’, and on the other hand also unacceptable delays of signals. (Commission

(2012)).

Correct crisis signals =

A
(A + C),

Type I error =

C
(A + C),

the share of crisis events, when the early warning system failed to give signal,

Type II error =

B
(B + D),

the share of events, when the early warning system incorrectly signalled crisis,

Noise-to-Signal Ratio =

B/(B + D)
A/(A + C))

,

the share of incorrectly signalled crises events and correct crisis signals.

Empirical evaluations of early warning indicators are frequently based on a usefulness function

’U’ proposed by Alessi and Detken (2011) that is based on a loss function L. In the loss function

certain weights θ and (1-θ) are assigned to Type I errors (C/(A + C)) and Type II errors (B/(B

+ D)). An early warning indicator is the more useful, the lower the NSR and the higher the

value of U is and is generally considered useful if U > 0.

Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C
(A + C)

− (1 − θ) ·

B
(B + D)

.

3.2 Diﬀerentiation of crisis events and of life-cycle phases

In this paper, the predictive power of the MIP Indicators are tested for all three phases of the crisis

life-cycle (Figure 3). The 1st PHASE is called ’Overheating’, which is often observed during the

run-up to crisis events. The 2nd PHASE is called ’Crisis’ and the 3rd PHASE is the ’Downturn’

in the aftermath of the crisis. Previous empirical studies on the MIP Scoreboard investigated only

whether its indicators could have forecast the 2nd and 3rd PHASE of crisis life-cycles, e.g. the

crisis events and downturns in EU Member States. This paper also tests whether MIP indicators

could help to signal the 1st PHASE of crisis events, the overheating of the economy (preventive

arm of the MIP).

8

The deﬁnition of crisis events vary across studies. Boysen et al. (2015) used a dataset of three

types of crisis (banking crises, currency crises, and debt crises) for the EU-28 countries and merged

these sets of dummies to construct a new dummy variable (labeled as ’ﬁnancial crisis dummy’ )

equaling one if at least one of these three types of crises occurs. Knedlik (2014) deﬁned its crisis

variable based on long-term government bond spreads.

We followed the practice of most empirical studies and considered a forecast horizon between

1 to 3 years (Borio et al. (2009), Knedlik (2014). Downturn and overheating in EU Member States

Figure 3: Possible and tested channels of the ’Signals Approach’

Macroeconomic Imbalance Procedure (MIP)

I. PHASE: Overheating
preventive arm of the MIP
not yet tested

II. PHASE: Crisis

Boysen-Hogrefe et al.(2015)
Knedlik (2014)

III. PHASE: Downturn
corrective arm of the MIP
Csortos-Szalai (2013, 2014)
Domonkos et al.(2017)

are deﬁned on the basis of the cyclical GDP gap similarly to earlier studies Domonkos et al. (2017);

Csortos and Szalai (2014) as follows. The real GDP of each country has been ﬁltered by the HP

(Hodrick–Prescott)-ﬁlter. To calculate the GDP gap, the diﬀerence between the basic time series

and trend has been taken. We determine the threshold of the critical diﬀerence at minus 2 per

cent for downturns, as proposed by the studies referred in this report. Macroeconomic overheating

situations are identiﬁed the same way, if the gap exceeded plus 2 percent.

3.3 Composite Indicators

We combine the signals approach of empirical studies and the composite indicator methodology

to provide a new method for evaluating the performance of the MIP Scoreboard. Such method

has been proposed by the empirical studies of Babecky et al. (2014), Erhart et al. (2018), Borio

et al. (2009), Christensen and Li (2014) to capture the fragility of the economy prior to a ﬁnancial

stress event most eﬃciently. Lo Duca et al. (2017) results on the newly established ﬁnancial crisis

database also conﬁrm that multivariate methods can improve upon univariate signaling models.

Babecky et al. (2014) note that a combination of several early warning indicators delivers a better-

performing early warning model compared to a single early warning predictor.

We deﬁned a composite measure as the sum of the red ﬂags (or signals) based on oﬃcial MIP

9

indicator  thresholds.  The  number  of  red  flags  has  already  been  incorporated  in  the  Commission 

analysis,  without  being  the  only  or  main  determinant  of  the  Alert  Mechanism  Report  outcomes 

Commission (2016).  The threshold for the total number of red flags has never been communicated 

yet  by  the  European  Commission.  We set  three  as  a  starting  value  for  the  threshold  of  the  com-

posite  measure  for  all  countries  in  each  year.  (Reminder:  on  average  four  indicators  were  flagged 

in  member  states  during  the  run-up  and  aftermath  of  the  2008-2009  crisis.)

4  Results

Our  key  result  is  that  the  usefulness  of  MIP  Scoreboard  Indicators’  signals  diﬀer  markedly  for  the 

diﬀerent kind of crisis events. The MIP Scoreboard Indicators gave relatively good signals for the (i) 

Currency/BoP/Capital  flow  crises  and  (ii)  Sovereign  crisis  events.  For  these  two  crisis  types, 

usefulness  of  best  MIP  indicators  is  in  the  range  between  0.2  and  0.3.  However,  in  case  of  (iii) 

Banking  crises  and  (iv)  Significant  asset  price  corrections  the  usefulness  of  MIP  indicators  is 

although mostly positive, but much smaller, around 0.1.

The  current  account  was  found  as  best  forecasting  indicator  for  all  crisis  events  (its  usefulness 

ranged  between  0.1  and  0.3).  Furthermore,  the  net  international  investment  position  was  also  a 

relatively  good  indicator  for  Currency/BoP/Capital  flow  and  Sovereign  crisis  events  (U  =  0.2) 

compared to other MIP indicators.  Frankel and Saravelos (2011) also found on the basis of a liter-

ature review of 83 papers that the current account balance is one of the most frequent statistically 

significant  indicators  in  explaining  crisis  incidence.

Earlier studies aimed at the identification of the events in the second half of the crisis life-cycle, e.g. 

crisis  events  and  ex-post  downturns.  Our  analysis  shows  that  some  MIP  indicators  can  signal  the 

first phase of the crisis life-cycle, e.g.  the economic overheating in EU Member States.  In particular 

the  real  house  price  and  total  financial  sector  liabilities  performed  well  in  this  setting.  Real  house 

prices  were  not  flagged  by  earlier  studies  amongst  best  indicators,  because  these  studies  focused 

on  the  ex-post  crisis  manifestation  indicators.  The  current  threshold  of  6  percent  annual  change 

can  help  to  identify  the  build-up  of  imbalance,  but  not  the  corrections  (see  further  discussion  on 

the  advantages  of  two-sided  MIP  thresholds  in  the  paper  of  Erhart  et  al.  (2018).)

The  usefulness  of  MIP  indicators  in  general,  is  higher  for  signalling  overheating  of  the  economy 

than that of downturns.  This finding suggests that the preventive arm of the MIP could be stronger 

than its corrective arm, which is perhaps a preferred policy choice as ’prevention is better than cure’.

10

Importantly,  the  simple  composite  of  threshold  breaches  performed  in  most  cases  as  good  as  the 

best  MIP  Scoreboard  indicator  in  the  baseline  scenario  (last  columns  in  Table  4).  Taking  into 

account  that  decisions  cannot  be  based  on  sole  indicators,  using  a  composite  MIP  measure  could

be  an  option  to  be  considered  by  decision  makers,  who  prefer  simple,  rule  based  decisions.

Table  3:  Summary  of  the  signals  approach  usefulness  metric  for  individual  indicators  and  the 
composite  (forecast  horizon  =  3  Y,  threshold  for  the  number  of  red  flags=  4,  rounded  to  one 
decimal  place)

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n
i

l
a
n
o
i
t
a
n
r
e
t
n
i

t
e
N
0.2
0.2
0.0
0.0
0.1
0.0

e
c
n
a
l
a
b

t
n
u
o
c
c
a

t
n
e
r
r
u
C
0.3
0.2
0.1
0.1
0.1
0.1

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

l
a
e
R
0.0
0.0
-0.1
-0.1
0.0
0.1

t
s
o
c

r
u
o
b
a
l

t
i
n
u

l
a
n
i
m
o
N
0.1
0.1
0.0
0.0
0.1
0.0

d
e
t
a
ﬂ
e
d

,
x
e
d
n
i

e
c
i
r
p

e
s
u
o
H
0.0
0.0
0.0
0.1
0.0
0.2

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

e
t
a
v
i
r
P
0.1
0.1
0.1
0.1
0.0
0.1

s
e
r
a
h
s

t
e
k
r
a
m

t
r
o
p
x
E
0.0
0.1
0.0
0.0
0.0
-0.1

t
b
e
d

r
o
t
c
e
s

e
t
a
v
i
r
P
0.0
0.1
0.1
0.0
0.0
0.0

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

l
a
r
e
n
e
G
0.1
0.1
0.1
0.1
0.0
0.0

e
t
a
r

t
n
e
m
y
o
l
p
m
e
n
U
0.0
0.0
-0.1
-0.1
0.0
0.0

)
3
>
s
g
a
ﬂ

d
e
r

f
o

o
N
(
E
T
I
S
O
P
M
O
C
0.3
0.2
0.1
0.1
0.1
0.1

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

l
a
i
c
n
a
n
ﬁ

l
a
t
o
T
0.1
0.0
0.0
0.1
0.0
0.2

r
o
t
a
c
i
d
n
i

P
I
M

t
s
e
B
0.3
0.2
0.1
0.1
0.1
0.2

Target variable
ESRB (Currency, BoP, Capital ﬂow)
ESRB (Sovereign)
ESRB (Banking)
ESRB (Signiﬁcant asset price correction)
Downturn (more severe than - 2pp.)
Overheating (min + 2pp.)

Our results are in line with the conclusions of the majority of earlier empirical studies on the MIP. 

They confirm that the definition of the crisis could be possible explanation for the diﬀerences in the 

empirical results. Csortos and Szalai (2014) and Domonkos et al. (2017) used the cyclical GDP gap, 

Boysen  et  al.  (2015)  analysed  the  union  of  financial  crisis  events  and  Knedlik  (2014)  debt  crisis 

events (Table 1.)

Knedlik  (2014)  investigated  debt  crisis  events  and  found  similarly  to  us  that  the  usefulness  is 

the highest for the current account, net international investment position and nominal unit labour 

costs  (Figure  4).

Earlier  empirical  studies  focused  on  specific  crisis  types,  when  measuring  the  predicting  power 

of  the  MIP  Scoreboard  indicators  and  there  have  not  been  any  attempt  to  analyse  the  impor-

tance  of  crisis  types  systemically  to  the  best  of  our  knowledge.  Boysen  et  al.  (2015)  focused  on 

the  union  of  financial  crisis  events  and  found  that  real  house  prices,  the  private  sector  credit  flow 

and  the  private  sector  debt  are  the  best  indicators  (Figure  5).  Our  results  are  similar  with  the 

exception of private sector debt, and that we found the current account a relatively good indicator.

11

If the MIP was used to signal any kind of crisis events the usefulness of indicators would be

in most cases close to or below 0.1. If the MIP Scoreboard was used to signal (i) Currency/BoP/-

Capital ﬂow crises and (ii) Sovereign crisis events the signalling power would improve and reach

the level of 0.2-0.3.

Figure 4: Usefulness of MIP Indicators to signal SOVEREIGN crisis events - Comparing our results
to Knedlik (2014)

Knedlik2014
Our results

Total ﬁnancial sector liabilities
Unemployment rate
General government sector debt
Private sector debt
Private sector credit ﬂow
House price index deﬂated
Nominal unit labour cost
Export market shares
Real eﬀective exchange rate
Net international investment position
Current account balance

0 5 · 10−2 0.1

0.15

0.2

0.25

0.3

Figure 5: Usefulness of MIP Indicators to signal UNION OF CRISIS events - Comparing our
results to Boysen et al. (2015)

Boysen et al. (2015)
Our results

Total ﬁnancial sector liabilities
Unemployment rate
General government sector debt
Private sector debt
Private sector credit ﬂow
House price index deﬂated
Nominal unit labour cost
Export market shares
Real eﬀective exchange rate
Net international investment position
Current account balance

−0.1−5 · 10−2 0

5 · 10−2 0.1

0.15

Csortos and Szalai (2014) also showed that the current account is the best indicator to forecast

downturns (it was the only MIP indicator, for which prediction ratios were better than the ratios

of false alarms to alarms total). According to our results, the current account, net international

investment position and nominal unit labour cost performed equally well as signalling indicators

(usefulness = 0.1).

12

Our key result that the MIP can presumably better capture macroeconomic problems than ﬁ-

nancial imbalances conﬁrms the ﬁndings of Detken et al. (2018) that simple credit and asset price

indicators have better early warning properties for domestic ﬁnancial crises in euro area countries

than macroeconomic indicators, such as the current account balance.

Most of our results are robust to the length of the forecast horizon (Table 4). The MIP could help

to recognize easier (i) Currency/BoP/Capital ﬂow crises and (ii) Sovereign crisis events. However,

the shorter the forecast horizon the better the predictive power of some indicators to identify the

overheating of the economy (current account, private sector credit ﬂow). Obviously, due to the

life-cycle and order of crisis events, macroeconomic imbalances start usually as an economic over-

heating, followed by crisis and later by downturn (See Figure 1). The export market share was a

relatively good indicator of (i) Currency/BoP/Capital ﬂow crises and (ii) sovereign crisis events if

the forecast horizon was set at 1 year.

5 Conclusions

Our key ﬁnding is that each unhappy crisis is ’unhappy in its own way’. Although it is very diﬃcult

to detect economic imbalances in advance, the MIP Scoreboard indicators could eﬀectively signal

(i) BoP/Currency/Capital ﬂow and (ii) Sovereign type of crisis events in our sample period be-

tween 2005 and 2016. They were less eﬀective, however, as signalling means in case of (iii) Banking

crises and (iv) Signiﬁcant asset price corrections.

We give further evidence that the current account is perhaps the most important indicator in

the MIP Scoreboard. Similarly to earlier studies, we ﬁnd it as the best forecasting indicator for all

crisis events (it’s usefulness ranged between 0.1 and 0.3). Furthermore, the net international in-

vestment position is also a relatively good indicator for Currency/BoP/Capital ﬂow and Sovereign

crisis events.

Previous empirical works tested the predictive power of indicators on the basis of events in the

second half of the crisis life-cycle, e.g. crisis events and ex-post downturns. Our analysis extended

the focus to the run-up to crisis events. We showed that some MIP indicators can signal the ﬁrst

phase of the crisis life-cycle, e.g. the overheating in EU Member States. In particular, the real

house prices, private sector credit ﬂow and total ﬁnancial sector liabilities. Earlier papers could not

have identiﬁed real house prices as good indicators, probably because these papers focused on the

ex-post crisis manifestation indicators. The current threshold of 6 percent annual change in real

13

house prices can help to identify the build-up of imbalances, but not the corrections (see further

discussion on the advantages of two-sided MIP thresholds in the paper of Erhart et al. (2018).)

Apparently, the relatively large Type I errors of real house prices (70-80% in all speciﬁcations

except for overheating) conﬁrm the challenge to identify macroeconomic imbalances.

We found that the ’usefulness’ metric of MIP indicators is usually higher for signalling over-

heating of the economy than that of downturns. This ﬁnding suggests that the preventive arm of

the MIP could be stronger than its corrective arm in line with the second principle of the MIP

Scoreboard ’the scoreboard (indicators and thresholds) are chosen as to provide a reliable signalling

device for potentially harmful imbalances and competitiveness loss at an early stage of their emer-

gence’ (Commission (2012)).

Due to the heterogeneity of crises, standardisation of their detection is obviously diﬃcult. Our

simple composite of MIP indicator threshold breaches performed in most cases as good as the

best MIP Scoreboard indicator. Therefore, further aggregation of scoreboard indicators could be

considered so as to gain a more simple, rule based and perhaps more accountable decision making

in the European Semester.

14

References

Alessi, L. and Detken, C. (2011). Quasi real time early warning indicators for costly asset price

boom/bust cycles: A role for global liquidity. European Journal of Political Economy, Vol. 27(3).

Alessi, L. and Detken, C. (2018).

Identifying excessive credit growth and leverage. Journal of

Financial Stability, Vol. 35.

Babecky, A., Havranek, T., Mateju, J., Rusnak, M., Smidkova, K., and Vasicek, B. (2014). Banking,

debt, and currency crises in developed countries. Journal of Financial Stability, Vol. 15.

Borio, C., Koban, A., and Drehmann, M. (2009). Assessing the risk of banking crises – revisited.

BIS Quarterly Review.

Boysen, A., Jansen, N., Poedt, M., and Schwarzmueller, T. (2015). An empirical evaluation of

macroeconomic surveillance in the European Union. Kiel Working Papers, Kiel Institute for the

World Economy.

Christensen, I. and Li, F. (2014). Predicting ﬁnancial stress events: a signal extraction approach.

Journal of Financial Stability, Vol. 14:54–65.

Commission, E. (2012). Scoreboard for the surveillance of macroeconomic imbalances. Occasional

Papers 92, doi: 10.2765/20196(92).

Commission, E. (2015). Adding employment indicators to the scoreboard of the macroeconomic im-

balance procedure to better capture employment and social developments, Ares (2015) 5426195.

na.

Commission, E. (2016). The macroeconomic imbalance procedure, rationale, process, application:

A compendium. Institutional Paper, doi:10.2765/958247(39).

Csortos, O. and Szalai, Z. (2014). Early warning indicators: ﬁnancial and macroeconomic imbal-

ances in central and eastern european countries. MNB Working Papers, Vol. 1(2/2014).

Detken, Fahr, S., and Lang, J. H. (2018). Predicting the likelihood and severity of ﬁnancial crises

over the medium term with a cyclical systemic risk indicator. Financial Stability Review.

Domonkos, T., Ostrihon, F., Sikulova, I., and Siranova, M. (2017). Analysing the relevance of the

MIP scoreboard’s indicators. National Institute Economic Review, Vol. 239(No 238).

Erhart, S., Becker, W., and Saisana, M. (2018). The Macroeconomic Imbalance Procedure - From

Thresholds to Decisions. JRC Technical Reports.

15

Frankel, J. and Saravelos, G. (2011). Can Leading Indicators Assess Country Vulnerability? Evi-

dence from the 2008-09 Global Financial Crisis. Faculty Research Working Paper Series.

Knedlik, T. (2014). The impact of preferences on early warning systems - The Case of the European

Commission’s Scoreboard. European Journal of Political Economy, Vol. 34:157–166.

Laeven, L. and Valencia, F. (2018). Systemic Banking Crises Revisited? IMF Working Paper No.

18/206.

Lo Duca, M., Koban, A., Barsten, M., Bengtsson, E., Klaus, B., Kusmierczyk, P., Lang, J. H.,

Detken, C., and Peltonen, T. (2017). A new database for ﬁnancial crises in European countries.

European Central Bank Occasional Paper Series No 194 .

16

A Appendix - Signal approach results for diﬀerent forecast

horizons

Table 4: Summary of the signals approach usefulness metric for individual indicators and the
composite (forecast horizon = 2 Y, threshold for the number of red ﬂags= 4, rounded to one
decimal place)

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n

i

l
a
n
o
i
t
a
n
r
e
t
n

i

t
e
N
0.2
0.2
0.0
0.0
0.1
0.0

e
c
n
a
l
a
b

t
n
u
o
c
c
a

t
n
e
r
r
u
C
0.2
0.2
0.1
0.1
0.0
0.1

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

l
a
e
R
0.0
0.0
-0.1
-0.1
0.0
0.0

t
s
o
c

r
u
o
b
a
l

t
i
n
u

i

l
a
n
m
o
N
0.1
0.1
0.0
0.0
0.1
0.0

d
e
t
a
ﬂ
e
d

,
x
e
d
n

i

e
c
i
r
p

e
s
u
o
H
0.0
0.0
0.0
0.0
-0.1
0.2

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

e
t
a
v
i
r
P
0.1
0.1
0.1
0.1
0.0
0.1

s
e
r
a
h
s

t
e
k
r
a
m

t
r
o
p
x
E
0.1
0.1
0.1
0.1
0.0
-0.1

t
b
e
d

r
o
t
c
e
s

e
t
a
v
i
r
P
0.0
0.0
0.0
0.0
0.0
0.0

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

l
a
r
e
n
e
G
0.1
0.1
0.1
0.1
0.0
0.0

e
t
a
r

t
n
e
m
y
o
l

p
m
e
n
U
0.1
0.1
-0.1
-0.1
0.0
0.0

)
3
>
s
g
a
ﬂ

d
e
r

f
o

o
N
(
E
T
I
S
O
P
M
O
C
0.3
0.2
0.1
0.1
0.0
0.1

s
e
i
t
i
l
i

b
a
i
l

r
o
t
c
e
s

l
a
i
c
n
a
n
ﬁ

l
a
t
o
T
0.0
0.0
0.0
0.0
-0.1
0.2

r
o
t
a
c
i
d
n

i

P
I
M

t
s
e
B
0.2
0.2
0.1
0.1
0.1
0.2

Target variable
ESRB (Currency, BoP, Capital ﬂow)
ESRB (Sovereign)
ESRB (Banking)
ESRB (Signiﬁcant asset price correction)
Downturn (more severe than - 2pp.)
Overheating (min + 2pp.)

Table 5: Summary of the signals approach usefulness metric for individual indicators and the
composite (forecast horizon = 1 Y, threshold for the number of red ﬂags= 4, rounded to one
decimal place)

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n
i

l
a
n
o
i
t
a
n
r
e
t
n
i

t
e
N
0.2
0.2
0.0
0.0
0.1
0.0

e
c
n
a
l
a
b

t
n
u
o
c
c
a

t
n
e
r
r
u
C
0.2
0.1
0.1
0.1
0.0
0.2

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

l
a
e
R
0.1
0.1
0.0
0.0
0.0
0.0

t
s
o
c

r
u
o
b
a
l

t
i
n
u

l
a
n
i
m
o
N
0.1
0.0
0.0
0.0
0.0
0.0

d
e
t
a
ﬂ
e
d

,
x
e
d
n
i

e
c
i
r
p

e
s
u
o
H
-0.1
-0.1
-0.1
-0.1
-0.1
0.2

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

e
t
a
v
i
r
P
0.1
0.0
0.0
0.1
-0.1
0.2

s
e
r
a
h
s

t
e
k
r
a
m

t
r
o
p
x
E
0.1
0.2
0.2
0.1
0.0
-0.1

t
b
e
d

r
o
t
c
e
s

e
t
a
v
i
r
P
0.0
0.0
0.0
0.0
0.0
0.0

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

l
a
r
e
n
e
G
0.1
0.1
0.1
0.1
0.0
-0.1

e
t
a
r

t
n
e
m
y
o
l
p
m
e
n
U
0.1
0.1
0.0
0.0
0.1
-0.1

)
3
>
s
g
a
ﬂ

d
e
r

f
o

o
N
(
E
T
I
S
O
P
M
O
C
0.3
0.2
0.1
0.1
0.0
0.1

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

l
a
i
c
n
a
n
ﬁ

l
a
t
o
T
0.0
-0.1
-0.1
-0.1
-0.1
0.2

r
o
t
a
c
i
d
n
i

P
I
M

t
s
e
B
0.2
0.2
0.2
0.1
0.1
0.2

Target variable
ESRB (Currency, BoP, Capital ﬂow)
ESRB (Sovereign)
ESRB (Banking)
ESRB (Signiﬁcant asset price correction)
Downturn (more severe than - 2pp.)
Overheating (min + 2pp.)

17

Table 6: Detailed statistics of the signals approach metrics for individual indicators and the
composite (crisis type: CURRENCY/BOP/Capital ﬂow, forecast horizon = 3 Y, threshold
for the number of red ﬂags= 4)

A - Correct signal (crisis signalled)
B - Wrong signal (NO crisis , signal)
C - Wrong signal (crisis, NO signal)
D - Correct signal (NO crisis, NO signal)
Correct forecasts (A+D)/(A+B+C+D)
Share of correct crisis signals A/(A+C)
type I error C/(A+C)
type II error B/(B+D)
NSR - Noise to signal ratio B/(B+D)/A/(A+C)
Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C

(A+C) − (1 − θ) ·

B
(B+D)

n
o
i
t
i
s
o
p

t
n
e
m

e
t
a
r

)
3
>
s
g
a
ﬂ

d
e
r

f
o

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

t
b
e
d

r
o
t
c
e
s

w
o
ﬂ

d
e
t
a
ﬂ
e
d

e
c
n
a
l
a
b

t
s
e
v
n
i

e
g
n
a
h
c
x
e

s
e
r
a
h
s

t
s
o
c

r
u
o
b
a
l

t
i
d
e
r
c

t
n
e
m
n
r
e
v
o
g

e
t
a
r

t
b
e
d

i

i

t
i
n
u

e
c
i
r
p

r
o
t
c
e
s

r
o
t
c
e
s

,
x
e
d
n
i

t
e
k
r
a
m

t
n
u
o
c
c
a

t
n
e
m
y
o
l

e
v
i
t
c
e
ﬀ
e

l
a
i
c
n
a
n
ﬁ

l
a
n
o
i
t
a
n
r
e
t
n

t
e
N
45
89
6
112

l
a
e
R
12
41
39
160

t
r
o
p
x
E
27
87
24
114

o
N
(
E
T
I
S
O
t
P
n
e
M
r
r
O
u
C
C
47
46
78
75
4
5
126
123
68% 62% 68% 56% 71% 69% 77% 56% 61% 67% 74% 67%
90% 88% 24% 53% 55% 27% 43% 49% 57% 31% 35% 92%
10% 12% 76% 47% 45% 73% 57% 51% 43% 69% 65% 8%
37% 44% 20% 43% 25% 21% 15% 42% 38% 24% 16% 39%
0.42
0.41
0.27
0.26
252
252

l
a
r
e
n
e
G
29
77
22
124

e
t
a
v
i
r
P
22
30
29
171

e
t
a
v
i
r
P
25
84
26
117

p
m
e
n
U
16
49
35
152

l
a
n
m
o
N
28
51
23
150

l
a
t
o
T
18
33
33
168

e
s
u
o
H
14
42
37
159

0.67
0.09
252

0.35
0.14
252

0.47
0.09
252

0.76
0.03
252

0.85
0.04
252

0.78
0.03
252

0.50
0.22
252

0.87
0.02
252

0.82
0.05
252

0.46
0.15
252

A
B
C
D
Correct forecasts
% of correct crisis signals
type I error
type II error
NSR
Usefulness
Total number of events

18

Table 7: Detailed statistics of the signals approach metrics for individual indicators and the
composite (crisis type: SOVEREIGN, forecast horizon = 3 Y, threshold for the number of red
ﬂags= 4)

A - Correct signal (crisis signalled)
B - Wrong signal (NO crisis , signal)
C - Wrong signal (crisis, NO signal)
D - Correct signal (NO crisis, NO signal)
Correct forecasts (A+D)/(A+B+C+D)
Share of correct crisis signals A/(A+C)
type I error C/(A+C)
type II error B/(B+D)
NSR - Noise to signal ratio B/(B+D)/A/(A+C)
Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C

(A+C) − (1 − θ) ·

B
(B+D)

n
o
i
t
i
s
o
p

t
n
e
m

e
t
a
r

)
3
>
s
g
a
ﬂ

d
e
r

f
o

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

t
b
e
d

r
o
t
c
e
s

w
o
ﬂ

d
e
t
a
ﬂ
e
d

e
c
n
a
l
a
b

t
s
e
v
n
i

e
g
n
a
h
c
x
e

s
e
r
a
h
s

t
s
o
c

r
u
o
b
a
l

t
i
d
e
r
c

t
n
e
m
n
r
e
v
o
g

e
t
a
r

t
b
e
d

i

i

t
i
n
u

e
c
i
r
p

r
o
t
c
e
s

r
o
t
c
e
s

,
x
e
d
n
i

t
e
k
r
a
m

t
n
u
o
c
c
a

t
n
e
m
y
o
l

e
v
i
t
c
e
ﬀ
e

l
a
i
c
n
a
n
ﬁ

l
a
n
o
i
t
a
n
r
e
t
n

t
e
N
41
93
7
111

l
a
e
R
10
43
38
161

t
r
o
p
x
E
27
87
21
117

o
N
(
E
T
I
S
O
t
P
n
e
M
r
r
O
u
C
C
38
37
87
84
10
11
120
117
62% 60% 68% 57% 69% 67% 75% 58% 60% 67% 69% 62%
77% 85% 21% 56% 50% 23% 38% 52% 54% 31% 21% 79%
23% 15% 79% 44% 50% 77% 63% 48% 46% 69% 79% 21%
41% 46% 21% 43% 27% 22% 17% 41% 39% 25% 20% 43%
0.54
0.53
0.18
0.18
252
252

l
a
r
e
n
e
G
26
80
22
124

e
t
a
v
i
r
P
18
34
30
170

e
t
a
v
i
r
P
25
84
23
120

p
m
e
n
U
15
50
33
154

l
a
n
m
o
N
24
55
24
149

l
a
t
o
T
10
41
38
163

e
s
u
o
H
11
45
37
159

0.72
0.07
252

0.44
0.10
252

0.96
0.00
252

0.79
0.05
252

1.01
0.00
252

0.76
0.07
252

0.53
0.20
252

0.96
0.00
252

0.54
0.12
252

0.78
0.03
252

A
B
C
D
Correct forecasts
% of correct crisis signals
type I error
type II error
NSR
Usefulness
Total number of events

19

Table 8: Detailed statistics of the signals approach metrics for individual indicators and the
composite (crisis type: BANKING, forecast horizon = 3 Y, threshold for the number of red
ﬂags= 4)

A - Correct signal (crisis signalled)
B - Wrong signal (NO crisis , signal)
C - Wrong signal (crisis, NO signal)
D - Correct signal (NO crisis, NO signal)
Correct forecasts (A+D)/(A+B+C+D)
Share of correct crisis signals A/(A+C)
type I error C/(A+C)
type II error B/(B+D)
NSR - Noise to signal ratio B/(B+D)/A/(A+C)
Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C

(A+C) − (1 − θ) ·

B
(B+D)

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n
i

l
a
n
o
i
t
a
n
r
e
t
n

e
c
n
a
l
a
b

t
n
u
o
c
c
a

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

t
s
o
c

r
u
o
b
a
l

t
i
n
u

s
e
r
a
h
s

t
e
k
r
a
m

d
e
t
a
ﬂ
e
d

,
x
e
d
n
i

e
c
i
r
p

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

t
b
e
d

r
o
t
c
e
s

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

e
t
a
r

t
n
e
m
y
o
l

l
a
i
c
n
a
n
ﬁ

i

i

l
a
n
m
o
N
34
45
65
108

t
e
N
49
85
50
68

t
r
o
p
x
E
47
67
52
86

e
t
a
v
i
r
P
49
60
50
93

l
a
r
e
n
e
G
51
55
48
98

l
a
e
R
12
41
87
112

e
s
u
o
H
28
28
71
125

e
t
a
v
i
r
P
33
19
66
134

t
n
e
r
r
u
C
57
64
42
89
58% 46% 49% 53% 56% 61% 66% 56% 59% 50% 60% 56%
58% 49% 12% 47% 34% 28% 33% 49% 52% 19% 24% 58%
42% 51% 88% 53% 66% 72% 67% 51% 48% 81% 76% 42%
42% 56% 27% 44% 29% 18% 12% 39% 36% 30% 18% 44%
0.77
0.73
0.07
0.08
252
252

p
m
e
n
U
19
46
80
107

l
a
t
o
T
24
27
75
126

1.57
-0.05
252

2.21
-0.07
252

1.12
-0.03
252

0.79
0.05
252

0.70
0.08
252

0.65
0.05
252

0.37
0.10
252

0.86
0.02
252

0.92
0.02
252

0.73
0.03
252

A
B
C
D
Correct forecasts
% of correct crisis signals
type I error
type II error
NSR
Usefulness
Total number of events

)
3
>
s
g
a
ﬂ

d
e
r

f
o

o
N
(
E
T
I
S
O
P
M
O
C
57
68
42
85

20

Table 9: Detailed statistics of the signals approach metrics for individual indicators and the
composite (crisis type: SIGNIFICANT ASSET PRICE CORRECTION, forecast horizon
= 3 Y, threshold for the number of red ﬂags= 4)

A - Correct signal (crisis signalled)
B - Wrong signal (NO crisis , signal)
C - Wrong signal (crisis, NO signal)
D - Correct signal (NO crisis, NO signal)
Correct forecasts (A+D)/(A+B+C+D)
Share of correct crisis signals A/(A+C)
type I error C/(A+C)
type II error B/(B+D)
NSR - Noise to signal ratio B/(B+D)/A/(A+C)
Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C

(A+C) − (1 − θ) ·

B
(B+D)

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n
i

l
a
n
o
i
t
a
n
r
e
t
n

e
c
n
a
l
a
b

t
n
u
o
c
c
a

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

t
s
o
c

r
u
o
b
a
l

t
i
n
u

s
e
r
a
h
s

t
e
k
r
a
m

d
e
t
a
ﬂ
e
d

,
x
e
d
n
i

e
c
i
r
p

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

t
b
e
d

r
o
t
c
e
s

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

e
t
a
r

t
n
e
m
y
o
l

l
a
i
c
n
a
n
ﬁ

i

i

l
a
n
m
o
N
37
42
65
108

t
e
N
51
83
51
67

t
r
o
p
x
E
47
67
55
83

e
t
a
v
i
r
P
49
60
53
90

l
a
r
e
n
e
G
51
55
51
95

l
a
e
R
15
38
87
112

e
s
u
o
H
30
26
72
124

e
t
a
v
i
r
P
35
17
67
133

t
n
e
r
r
u
C
60
61
42
89
59% 47% 50% 52% 58% 61% 67% 55% 58% 49% 61% 58%
59% 50% 15% 46% 36% 29% 34% 48% 50% 19% 26% 59%
41% 50% 85% 54% 64% 71% 66% 52% 50% 81% 74% 41%
41% 55% 25% 45% 28% 17% 11% 40% 37% 31% 16% 43%
0.74
0.69
0.08
0.09
252
252

p
m
e
n
U
19
46
83
104

l
a
t
o
T
27
24
75
126

1.65
-0.06
252

1.72
-0.05
252

1.11
-0.03
252

0.83
0.04
252

0.73
0.07
252

0.59
0.06
252

0.77
0.04
252

0.33
0.11
252

0.97
0.01
252

0.60
0.05
252

A
B
C
D
Correct forecasts
% of correct crisis signals
type I error
type II error
NSR
Usefulness
Total number of events

)
3
>
s
g
a
ﬂ

d
e
r

f
o

o
N
(
E
T
I
S
O
P
M
O
C
60
65
42
85

21

Table 10: Detailed statistics of the signals approach metrics for individual indicators and the
composite (crisis type: DOWNTURN (NEGATIVE GDP GAP (-2%)), forecast horizon =
3 Y, threshold for the number of red ﬂags= 4)

A - Correct signal (crisis signalled)
B - Wrong signal (NO crisis , signal)
C - Wrong signal (crisis, NO signal)
D - Correct signal (NO crisis, NO signal)
Correct forecasts (A+D)/(A+B+C+D)
Share of correct crisis signals A/(A+C)
type I error C/(A+C)
type II error B/(B+D)
NSR - Noise to signal ratio B/(B+D)/A/(A+C)
Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C

(A+C) − (1 − θ) ·

B
(B+D)

n
o
i
t
i
s
o
p

t
n
e
m

t
s
e
v
n
i

l
a
n
o
i
t
a
n
r
e
t
n

e
c
n
a
l
a
b

t
n
u
o
c
c
a

e
t
a
r

e
g
n
a
h
c
x
e

e
v
i
t
c
e
ﬀ
e

t
s
o
c

r
u
o
b
a
l

t
i
n
u

s
e
r
a
h
s

t
e
k
r
a
m

d
e
t
a
ﬂ
e
d

,
x
e
d
n
i

e
c
i
r
p

t
b
e
d

r
o
t
c
e
s

t
n
e
m
n
r
e
v
o
g

w
o
ﬂ

t
i
d
e
r
c

r
o
t
c
e
s

t
b
e
d

r
o
t
c
e
s

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

e
t
a
r

t
n
e
m
y
o
l

l
a
i
c
n
a
n
ﬁ

i

i

l
a
n
m
o
N
46
33
51
122

t
e
N
62
72
35
83

t
r
o
p
x
E
43
71
54
84

e
t
a
v
i
r
P
46
63
51
92

l
a
r
e
n
e
G
36
70
61
85

l
a
e
R
23
30
74
125

e
s
u
o
H
17
39
80
116

e
t
a
v
i
r
P
18
34
79
121

t
n
e
r
r
u
C
57
64
40
91
59% 58% 59% 50% 67% 53% 55% 55% 48% 53% 53% 57%
59% 64% 24% 44% 47% 18% 19% 47% 37% 23% 15% 59%
41% 36% 76% 56% 53% 82% 81% 53% 63% 77% 85% 41%
41% 46% 19% 46% 21% 25% 22% 41% 45% 28% 23% 44%
0.75
0.70
0.07
0.09
252
252

p
m
e
n
U
22
43
75
112

l
a
t
o
T
15
36
82
119

1.44
-0.04
252

1.22
-0.03
252

1.18
-0.02
252

1.03
-0.01
252

1.50
-0.04
252

1.22
-0.04
252

0.45
0.13
252

0.86
0.03
252

0.82
0.02
252

0.73
0.09
252

A
B
C
D
Correct forecasts
% of correct crisis signals
type I error
type II error
NSR
Usefulness
Total number of events

)
3
>
s
g
a
ﬂ

d
e
r

f
o

o
N
(
E
T
I
S
O
P
M
O
C
57
68
40
87

22

Table 11: Detailed statistics of the signals approach metrics for individual indicators and the
composite (crisis type: OVERHEATING (POSITIVE GDP GAP (+2%)), forecast horizon
= 3 Y, threshold for the number of red ﬂags= 4)

A - Correct signal (crisis signalled)
B - Wrong signal (NO crisis , signal)
C - Wrong signal (crisis, NO signal)
D - Correct signal (NO crisis, NO signal)
Correct forecasts (A+D)/(A+B+C+D)
Share of correct crisis signals A/(A+C)
type I error C/(A+C)
type II error B/(B+D)
NSR - Noise to signal ratio B/(B+D)/A/(A+C)
Usefulness = min(θ; 1 − θ) − L = min(θ; 1 − θ) − θ ·

C

(A+C) − (1 − θ) ·

B
(B+D)

n
o
i
t
i
s
o
p

t
n
e
m

e
t
a
r

)
3
>
s
g
a
ﬂ

d
e
r

f
o

s
e
i
t
i
l
i
b
a
i
l

r
o
t
c
e
s

t
b
e
d

r
o
t
c
e
s

w
o
ﬂ

d
e
t
a
ﬂ
e
d

t
s
o
c

r
u
o
b
a
l

e
c
n
a
l
a
b

t
s
e
v
n
i

e
g
n
a
h
c
x
e

s
e
r
a
h
s

t
i
d
e
r
c

t
n
e
m
n
r
e
v
o
g

e
t
a
r

t
b
e
d

i

i

t
i
n
u

e
c
i
r
p

r
o
t
c
e
s

r
o
t
c
e
s

,
x
e
d
n
i

t
e
k
r
a
m

t
n
u
o
c
c
a

t
n
e
m
y
o
l

e
v
i
t
c
e
ﬀ
e

l
a
i
c
n
a
n
ﬁ

l
a
n
o
i
t
a
n
r
e
t
n

t
e
N
22
112
19
99

l
a
e
R
13
40
28
171

t
r
o
p
x
E
9
105
32
106

o
N
(
E
T
I
S
O
t
P
n
e
M
r
r
O
u
C
C
29
25
96
96
12
16
115
115
56% 48% 73% 46% 63% 77% 75% 54% 55% 68% 83% 57%
61% 54% 32% 22% 32% 49% 37% 41% 41% 32% 61% 71%
39% 46% 68% 78% 68% 51% 63% 59% 59% 68% 39% 29%
45% 53% 19% 50% 31% 17% 18% 44% 42% 25% 12% 45%
0.64
0.75
0.13
0.08
252
252

l
a
r
e
n
e
G
17
89
24
122

e
t
a
v
i
r
P
15
37
26
174

e
t
a
v
i
r
P
17
92
24
119

p
m
e
n
U
13
52
28
159

l
a
n
m
o
N
13
66
28
145

l
a
t
o
T
25
26
16
185

e
s
u
o
H
20
36
21
175

2.27
-0.14
252

1.05
-0.01
252

0.35
0.16
252

0.78
0.04
252

0.20
0.24
252

1.02
0.00
252

0.48
0.10
252

0.99
0.00
252

0.99
0.00
252

0.60
0.06
252

A
B
C
D
Correct forecasts
% of correct crisis signals
type I error
type II error
NSR
Usefulness
Total number of events

23

Imprint and acknowledgements 

The author gratefully acknowledges the help, comments and suggestions received from Zsuzsa Turóczy (ex 
Trafo), Anna Naszódi (Joint Research Centre, European Comission), Michaela Saisana (Joint Research Centre, 
European Comission), Daniel Vértesy (Joint Research Centre, European Comission), Lucia Alessi (Joint 
Research Centre, European Comission), Sven Langedijk (Joint Research Centre, European Comission), 
Claudia Ghisetti (Joint Research Centre, European Comission), from Jean-Charles Bricongne (DG ECFIN, 
European Commission), from Carsten Detken (European Systemic Risk Board), Tuomas Peltonen (European 
Systemic Risk Board). The views expressed in this paper are those of the author and do not necessarily reflect 
the official views of the European Commission. 

Szilard Erhart 
European Commission, Brussels, Belgium; email: szilard.erhart@ec.europa.eu 

© European Systemic Risk Board, 2019 

Postal address 
Telephone 
Website 

60640 Frankfurt am Main, Germany 
+49 69 1344 0 
www.esrb.europa.eu  

All rights reserved. Reproduction for educational and non-commercial purposes is permitted provided that the 
source is acknowledged. 

Note: 
The views expressed in ESRB Working Papers are those of the authors and do not necessarily reflect 
the official stance of the ESRB, its member institutions, or the institutions to which the authors are 
affiliated. 

ISSN 
ISBN 
DOI 
EU catalogue No 

2467-0677 (pdf) 
978-92-9472-116-7 (pdf) 
10.2849/681425 (pdf) 
DT-AD-19-016-EN-N (pdf)