Claudia M. Buch
Vice-President of Deutsche Bundesbank
Policy Evaluation:
Assessing the Effects of Post-Crisis Financial Sector
Reforms
Keynote prepared for the workshop
Women in Macro, Finance, and Economic History
DIW Berlin
June 17-18, 2019
Seite 1 von 13
Deutsche Bundesbank, Directorate General Communications
Wilhelm-Epstein-Strasse 14, 60431 Frankfurt am Main, Germany, Tel: +49 (0)69 9566 3511 or 3512, Fax: +49 (0)69 9566 3077
presse@bundesbank.de, www.bundesbank.de
Reproduction permitted only if source is stated.
The global financial crisis has been a watershed for the global economy. Its impli-
cations for growth, distribution, and public finances have been severe. Structural
changes have affected both the real economy and the financial sector. Reforms of
the financial regulatory system have not only been a consequence, but also a driv-
er of these trends. The G20 countries undertook an internationally coordinated
overhaul of the financial regulatory landscape. It has been the broad objective of
these reforms to reduce the probability and impact of future financial crises.
The crisis has particularly challenged the perception that the stability of individual
institutions automatically guarantees the stability of the entire financial system.
Macroprudential now complements microprudential supervision. The purpose of
macroprudential policy is to ensure that the financial system can perform its role for
the real economy, even in times of crises or distress.
In the following, I will briefly review the origins of financial crises and the im-
portance of taking a systemic view. I will then review the reform agenda of the past
10 years. As the implementation of key reform elements is well underway, the Fi-
nancial Stability Board (FSB) has started a structured approach towards ex post
evaluation as an integral part of a policy cycle. Ex post policy evaluation assesses
whether reforms have achieved their objectives and whether there have been ma-
jor unintended side effects. I will conclude by discussing elements of an evaluation
process which ensure independence and transparency.
1) What causes financial crises?
A stable financial system is consistently in a position to absorb both financial and
real economic shocks, especially when confronted with unforeseen events, in
stress situations and in periods of structural adjustment. A resilient financial system
is able to absorb losses from unexpected developments, it prevents contagion and
adverse feedback effects.
Financial stability is at risk if systemic risks emerge. At the heart of systemic risks
are externalities: Systemic risk may occur, for instance, when the distress of one or
more market participants jeopardises the functioning of the entire system. This
may be the case when the distressed market participant is very large or closely in-
terlinked with other market players.
Interconnectedness can be a channel through which unexpected adverse devel-
opments are transmitted to the financial system as a whole, thus impairing its sta-
bility. Market participants can be connected directly through contractual relation-
ship such as on the interbank market. Besides this, indirect channels of contagion
may exist. Many market participants may conduct similar transactions or hold simi-
lar types of assets. In such a situation, investors may interpret negative news as
signals that other market participants are adversely affected as well. Hence, sys-
temic risks also exist if a large number of small market participants is exposed to
similar risks or risks that are closely correlated with each other.
In the presence of systemic risk externalities, the optimal level of risk taken by fi-
nancial institutions from a private perspective is thus higher than the socially effi-
cient level of risk. Owners of financial institutions may reap the potential benefits of
their decisions in full, but bear the potential risks only partially if they can expect to
be bailed out by the government bailout in the event of distress.
This excessive risk taking implies costs for society: if risks materialize, govern-
ments may be pressured to bail out banks, thus imposing costs on taxpayers.
Regulatory reforms aimed at internalizing these externalities comprise higher loss-
absorbing capacity and provisions to facilitate the orderly resolution of banks.
These reforms have the potential to increase social welfare. However, they may
come at a net private cost for the affected institutions: they will forgo the net private
benefit (profits) associated with their excessive risk taking, they may incur costs re-
lated to compliance with new regulations, and their funding may be more costly as
it must provide greater capacity to absorb losses. Through these channels, chang-
es in regulations affect the incentives for banks to take on risks.
2) What happened since the financial crisis?
Capital flow/default cycles have been around since at least 1800 - if not before.
Technology has changed, the height of humans has changed, and fashions have
changed. Yet the ability of governments and investors to delude themselves, giving
rise to periodic bouts of euphoria that usually end in tears, seems to have re-
mained a constant. (Kaminsky and Rogoff 2008, p. 53)1.
Is this time different? Have policymakers learned from the recent financial crisis?
One key insight of the financial crisis has been that supervision needs to go be-
yond the individual institution and take a system-wide perspective. Macroprudential
policy aims to mitigate systemic risks, i.e. risks to the financial system as a whole.
It builds on microprudential regulations which are concerned with the stability of in-
dividual financial institutions. The need for a macroprudential perspective is a les-
son learned from the global financial crisis: even though individual financial institu-
tions may seem stable from a microprudential perspective, the financial system
may be highly vulnerable to shocks, and it may amplify these shocks.
Since 2009, the G20 has implemented a wide range of policies which aim at reduc-
ing the probability and the effects of future financial crises (Figure 1). These fall in-
to four areas:2 A first set of policies aims at enhancing the resilience of the finan-
cial sector with regard to adverse shocks. These policies include higher capital and
liquidity requirements for all banks and in particular for systemically important
banks. More resilient financial institutions can better withstand negative shocks
and are less likely to amplify them. This mitigates negative repercussions on the
real economy.
One key insight of the financial crisis has been that large, complex, and connected
financial institutions can be particularly systemic and "too big to fail". Large and
capital requirements, and these banks are also in the focus of new resolution poli-
cies. Requirements for their total loss-absorbing capacity (TLAC)3 are intended to
ensure that systemically important banks have issued a sufficient amount of bail-
inable debt and can be resolved without recourse to taxpayer money, while main-
taining their vital economic functions. Recovery and resolution planning identifies
critical functions that can be ring-fenced and shielded from the negative conse-
quences of adverse scenarios. Finally, enhanced supervision of systemically im-
portant banks includes higher supervisory expectations for risk management func-
tions, risk governance, and internal controls.
Additional policies aim at making derivatives markets safer and more transparent.
They include requirements on trade reporting, central clearing, platform trading,
margins and higher capital requirements for non-centrally cleared derivatives. Their
broader aim is to mitigate systemic risk, contagion, and to avoid market abuse. A
final set of policies aims at transforming shadow banking into resilient market-
based finance. These policies include new regulations for money market funds and
securitizations. Their broad purpose is to mitigate systemic risks associated with
non-bank financial intermediation.
3) What are the effects of the reforms?4
Many post-crisis financial regulatory reforms have entered uncharted territory: reli-
able assessments of reform effects and knowledge about transmission channels
were lacking. Reforms had to be implemented in an environment characterized by
a high degree of uncertainty. Hence, a structured policy cycle is needed to reduce
the risks of ineffective policies, miscalibrated policy instruments, and unintended
side effects. Such a cycle involves four steps (Figure 2):
(1) defining policy objectives,
(2) choosing intermediate objectives and appropriate indicators,
(3) linking instruments to indicators through ex ante impact assessments, and
(4) analysing policy effects through ex post evaluations.
In a first step, the policy objective(s) of macroprudential policy need to be speci-
fied. Macroprudential authorities use different definitions of the policy objective, but
they all aim at reducing systemic risk arising from externalities. Ideally, the policy
objective is derived from an analytical framework which links performance of the fi-
nancial system to the functioning of the real economy.
In a second step, intermediate objectives need to be specified and appropriate in-
dicators chosen. Intermediate objectives are linked to the drivers of systemic risk,
such as leverage, risk-taking incentives, interconnectedness, or exposure to com-
mon shocks. In many cases, it will not be possible to specify a direct, simple, and
linear relationship between intermediate objectives and financial stability. For ex-
ample, the link between the interconnectedness of financial institutions and the
probability of financial contagion is non-linear.5 Intermediate objectives can thus
be state-dependent as their relationship to financial stability depends on the struc-
ture and state of the financial system and real economy.
In a third step, the activation or recalibration of policy instruments that address
evolving vulnerabilities and financial stability risks needs to be considered. The de-
cision on whether and how to activate policy measures should be based on a
structured process of ex ante policy evaluation. This provides information about the
relative contribution of policy instruments reducing systemic risk. Trade-offs be-
tween stability and other policy objectives can be taken into account when per-
forming such an analysis and when calibrating instruments.
In a fourth step, and once sufficient time has elapsed, the effects of the instru-
ments need to be assessed in an ex post evaluation. This step provides infor-
mation about the effectiveness of the measure(s) taken, about intended or unin-
tended side effects, and it also serves as an input into a possible recalibration of
the policy instruments.
Ultimately, policy evaluation provides insights beyond the assessment of specific
policy instruments by answering the question of "how to get there". Indeed, it can
also help address the question of "where to go" by providing information about the
appropriateness of the specific policy objectives and targets, which may be revised
if needed. This is particularly important given that macroprudential policy is a rela-
tively new area and given that experience with the above policy cycle is limited.
Evaluation of financial sector policies: The approach of the Financial Stabil-
ity Board
The Financial Stability Board (FSB) has recently developed a framework to guide
its own evaluation work related to the G20 post-crisis reforms of financial sector
regulation.6 According to the FSB Framework, evaluation methods need to answer
three questions:
• Did the reform "cause" an outcome (attribution)?
• Did the reform have similar effects across relevant markets, states of the
world, or jurisdictions and regions (heterogeneity)?
• Did the reform achieve its overall objective (aggregation/general equilibri-
um)?
Analytical tools vary in their ability to answer these questions. Which evaluation
approach to choose depends on their relevance to the objectives of the evaluation,
their feasibility and replicability for implementation across countries, and their costs
in terms of resource inputs and data requirements. As shown by the first FSB eval-
uations, a robust evaluation strategy combines different (both qualitative and quan-
titative) analytical approaches.
In addition to fleshing out analytical concepts, the FSB framework contains a num-
ber of arrangements concerning the governance of evaluation work. For instance,
the framework recommends that the evaluation process should include means for
engagement with external stakeholders. These may include the publication of in-
formation on forthcoming evaluation projects, public consultations on evaluation
reports before publication, and the publication of evaluation reports. Workshops
and roundtables of FSB members with academia and participants from the industry
and the civil society are means of interaction.
The first two evaluations under the framework have been delivered to the G20-
Summit in November 2018:7
• The first project investigates the extent to which post-crisis reforms incentiv-
ized the central clearing of OTC derivatives. It has been the aim of these reforms
to make derivatives markets both, safer and more transparent. Evaluation results
suggest that reforms are providing the intended incentives for central clearing, es-
pecially for the most systemic market participants.
• The second project evaluates the effect of reforms on the financing of infra-
structure. It concluded that effects of the reforms on infrastructure finance were of
second order importance relative to other factors such as the macro-financial envi-
ronment, fiscal and monetary policies. The analyses do not point to a significant ef-
fect on the volume and cost of infrastructure finance, while there are some indica-
tions that reforms may have shortened the average maturities of infrastructure
loans granted by global systemically important banks.
More projects are underway. An evaluation of reform effects on SME financing will
be delivered to the G20 Summit this year. The respective consultation report was
published recently.8 According to this report, the evaluation does not identify per-
sistent negative effects of the G20 regulatory reforms on SME financing overall. At
the same time, the adjustment patterns in the data are not inconsistent with
stronger effects of the reforms on those banks for which the new rules on capital
regulation have been binding the most.
A new project which has been started in 2019 focuses on a core objective of the
reforms. This project will assess the extent to which the too-big-to fail (TBTF) re-
forms have addressed the systemic and moral hazard risks associated with sys-
temically important financial institutions:9
• The project will analyse whether the reforms were effective in reducing the
perceived probability of the impact of failure of systemically important institutions -
for instance, by assessing changes in estimates of implicit funding subsidies and
analysing progress in the resolvability of systemically important banks.
• It will provide answers to the question of whether reforms are leading to
changes in bank behaviour, for instance by attributing changes in business models
and risk taking that reflect the internalization of the negative systemic risk externali-
ties to relevant reforms.
• It will examine the broader effects of the reforms, including the functioning of
the financial system, market fragmentation, cost and availability of financing to the
real economy. A key issue will be to focus on the achievement of intended out-
comes but also analyse potential (positive or negative) unintended consequences.
The FSB, in line with its aim to engage with a broad range of stakeholders, has in-
vited feedback on this project, including supporting evidence of reform effects, by
June 21, 2019. To put the evaluation on a solid analytical basis, academic advisors
are working in close collaboration with the group on methodological issues.10
Stakeholders will have further opportunities to interact with the FSB and contribute
to this project: as the project progresses, workshops and roundtables will provide
an opportunity for externals to engage with the FSB evaluation team. Lastly, the
FSB plans to publish a consultation report in June 2020.
4) Independence of policy evaluations11
Credible policy evaluation obviously requires independence. Independent and ob-
jective assessments are needed to gain an unbiased picture of the effects of re-
forms. There are several avenues leading to this goal. External experts can be in-
volved who have a mandate for independent evaluations and who have no stake in
the formulation of policies. These can be existing institutions such as research in-
stitutes or policy advisory bodies. Independence of internal evaluations can be en-
hanced through institutional arrangements such as peer reviews or independence
from policy groups.
Transparency is another precondition for a credible regulatory process. Policy
evaluation means being transparent about the goals of regulatory policies and
what these policies have actually achieved. Repositories of evaluation studies can
improve transparency by making it easier and less costly for (internal) evaluators
and external stakeholders to keep track of evaluation work. Other fields, such as
development economics or medicine, often rely on repositories to structure the
available evidence.12 The Bank for International Settlements (BIS) recently
launched an online repository of studies on the effects of financial regulations.13
The repository is an interactive tool, allowing the user to select and visualize find-
ings from multiple studies. Beyond this, it also allows users to submit their own
studies.
Replicability is an important element of transparency. Replicability ensures that
policy decisions are based on robust evidence. This requires transparency about
the models being used, access to data, and documentation of how studies are set
up. Replication studies repeat an original analysis using different data and are thus
especially useful for policy work. Academic study often focuses on specific novel
results, which may limit the external validity of findings in a different setting. Evi-
dence-based policy decisions require information on whether a particular finding is
robust with regard to changing the underlying setting. Hence, there is a potential
tension between the novelty and uniqueness of findings that are appreciated in ac-
ademia and the need to rely on sufficiently robust results as a basis for policy deci-
sions. Enhanced appreciation of replication studies in academic work and en-
hanced capacity of policy institutions to conduct their own replication studies can
help relaxing this tension.
Finally, an active outreach strategy should also be part of an independent, trans-
parent evaluation process. Stakeholders should be given the opportunity to give
input and voice their concerns. This interaction can take place via public consulta-
tions of reports, through workshops or conferences, or by appointing external aca-
demic advisors. All this will ultimately improve the quality of evaluations.
Figure 1: Post-Crisis Financial Sector Reforms
Source: Cavallo, Anne-Sophie (2018), "Evaluating the impact of international fi-
nancial reforms", Banque de France Bulletin, Spring 2018.
Seite 12 von 13
Deutsche Bundesbank, Directorate General Communications
Wilhelm-Epstein-Strasse 14, 60431 Frankfurt am Main, Germany, Tel: +49 (0)69 9566 3511 or 3512, Fax: +49 (0)69 9566 3077
presse@bundesbank.de, www.bundesbank.de
Reproduction permitted only if source is stated.
Figure 2: Stylized representation of a macroprudential policy process
Source: Buch, Vogel, Weigert (2018)
Seite 13 von 13
Deutsche Bundesbank, Directorate General Communications
Wilhelm-Epstein-Strasse 14, 60431 Frankfurt am Main, Germany, Tel: +49 (0)69 9566 3511 or 3512, Fax: +49 (0)69 9566 3077
presse@bundesbank.de, www.bundesbank.de
Reproduction permitted only if source is stated.