Claudia Buch: Evaluating financial sector reforms - a joint task for
academia and policymakers
Statement by Prof Claudia Buch, Deputy President of the Deutsche Bundesbank, for the panel
discussion "Improving Financial Resilience"; at the T20 Summit "Global Solutions", Berlin, 30
May 2017.
* * *
Since the global financial crisis, major reforms have been launched which aim to enhance the
stability of the financial system, making future systemic crises less likely and reducing collateral
damage in the real economy. Evaluating the effects of these reforms needs to be part of a
structured process that answers three questions: Did the reform cause an outcome? Have
reforms had similar effects across markets and jurisdictions? Has the reform achieved its overall
objective? Economic research has advanced significantly over the past decades and provides a
rich infrastructure that can be used to answer these questions. Based on transparency,
international coordination and independent assessments, academia and policymakers can join
forces to conduct good policy evaluations.
I. The long shadows of financial crises
Ten years after the onset of the global financial crisis, its legacies still weigh heavily on the
financial sector and the real economy. The crisis emerged from the core of the global financial
system, but it has had significant economic and social costs globally. Output losses occurred,
wealth decreased, unemployment went up - with all its social consequences - and public debt
increased (Figure 1).
The Financial Stability Board (FSB) estimates that the cumulative output loss since the global
financial crisis is in the order of 25 % of one year's world GDP (FSB 2016). Taking a longer-term
perspective for the years 1970-2011 shows that financial crises were followed by output losses
relative to GDP of 26 % for emerging economies and 33 % for advanced economies (Laeven and
Valencia 2012). While the exact magnitude of the decline depends on country characteristics and
the way the crisis is measured, these numbers confirm that costs of financial crises are
immense and long-lasting.
Since the global financial crisis, major reforms have thus been launched that aim to enhance the
resilience of the financial system, making future systemic crises less likely and reducing damage
done to the real economy. Financial market participants should only take risks if they are able to
bear the potential losses without taking recourse to public funds.
Regulations have been amended to strengthen the private sector's capacity to buffer shocks, to
reduce implicit public subsidies for the financial sector, and to promote strong, sustainable, and
inclusive growth. G20 leaders have identified four main reform areas: making banks more
resilient, ending too-big-to-fail, transforming shadow banking into market based finance, and
making derivative markets safer.
Many financial sector reforms have entered uncharted territory and their impact still needs to be
assessed. Because the implementation of many core reforms is well underway (FSB 2016), an
initial analysis is now becoming possible. The FSB thus developed a Framework for the
PostImplementation Evaluation of the Effects of the G20 Financial Regulatory Reforms. At their
meeting in Baden-Baden, Germany, in March 2017, the G20 reaffirmed their commitment to a
structured evaluation of these reforms and the FSB published the draft framework for a public
consultation (FSB 2017). The framework will be submitted to the G20 Leaders' Summit in
Hamburg for endorsement and will subsequently be published.
II. Embedding evaluation in a structured policy process
Policy evaluation needs to be part of a structured policy process, which involves four steps:
Step 1: Specify the objectives of reforms
The objectives of reforms need to be defined as a benchmark against which progress can be
measured. The paramount goal of financial sector reforms has been to realign private incentives,
whereby private actors are to bear the costs of financial distress in the same way they might
reap the economic benefits. At least two dimensions must be considered. According to a narrow
view, goals of specific reform areas can be defined, such as the ability of individual banks to
absorb shocks arising from financial and economic stress or to improve risk management and
governance. From a broader perspective that looks at financial stability, the objectives might
target the stability of the financial system as a whole, the orderly functioning of markets, the
cost/availability of financing, and economic growth.
Step 2: Define intermediate targets
It is only possible to ascertain whether reforms are effective over a longer period of time. In the
interim, one has to rely on a range of intermediate variables for monitoring. For more narrowly
defined policy objectives, this includes measures of risk-based capital, liquidity and leverage as
well as evidence of improved bank governance, risk management, or reporting practices. For
broader objectives, one can use standard indicators of financial system resilience, the cost/level
of financing, or market liquidity.
Step 3: Calibration of instruments and ex ante impact assessment
Once the policy objectives and intermediate targets have been established, instruments need to
be calibrated accordingly. This requires an ex ante impact assessment.
Step 4: Ex post impact assessment
In a final step, once instruments have been applied and the effects have been observed over a
sufficiently long period of time, the impact of reforms can be assessed ex post.
Currently, such a framework for the evaluation of post-crisis financial sector reforms is lacking at
the global level. This is the gap that the FSB framework is about to close. It is about answering
the question of whether the reforms have achieved their intended outcomes, whether they work
together as intended, and whether they have had material unintended consequences. Such side
effects may have to be addressed, but without compromising on the objectives of the reforms or
by reducing resilience. The framework provides a common understanding of the elements
required for a "good" policy evaluation, and it will provide a basis for an informed and
evidencebased discussion on regulatory policies.
Policy evaluation need not be designed from the ground up. In many policy areas, structured
evaluations are routinely applied. Labor market policies and educational programs are often
implemented following a testing phase, and evaluations are commonly applied in the area of
2
development policies. Like clinical drug trials in medicine, policy evaluation can help to analyze
whether a policy, or "treatment", is effective, whether it works on the causes of the "disease" or
whether it merely relieves its symptoms. It can also help to identify any potential side effects.
"How do regulations affect the stability of financial institutions?" is similar to asking "How does
this drug affect the health of the patient?".
III. Addressing challenges of policy evaluation
Policy evaluation is challenging, as it requires dealing with attribution, heterogeneity, and general
equilibrium effects. Embedding evaluation into a structured policy process makes these
challenges explicit and makes it possible to address them. Transparency, international
cooperation, and independent assessments are key elements of such a process.
a) Did the reform cause an outcome (attribution)?
Disentangling the costs and benefits of regulation is the key challenge of any evaluation.
Financial reforms generate direct compliance costs for market participants, such as the costs of
reporting requirements, and these costs tend to be recognized immediately. The benefits of a
more stable and resilient financial system, by contrast, can be reaped only over the longer term
and are more difficult to quantify.
The direct compliance costs need to be distinguished from costs of crises that are borne by
market participants. The bailout of banks during the crisis shifted private costs to taxpayers and
raised levels of public debt worldwide. Many reforms thus seek to realign incentives by
withdrawing implicit public guarantees. As a result, risk-taking on the part of banks may decline,
allocation of credit may change, and costs of debt may increase because creditors cannot rely
on bailouts and thus demand higher risk premia. Profits in the financial sector may decline as a
consequence of reforms.
Taken together, costs are shifted from the public sector to the (private) financial sector. An
example from environmental policy may serve to illustrate the point. A Pigovian tax on chemical
producers that pollute a river internalises the externalities of polluting the river by aligning private
and social costs, thereby realigning the incentives to pollute. The direct result of this policy is
lower production of chemicals, as pollution of the river is now part of the private sector's costs,
and lower sectoral profits. However, these are not the costs of the reform but the intended
consequence.
Practical implementation of reform assessments needs to address the issue of a missing
counterfactual. How should the benefits of the reforms that have been implemented be
measured ex post? Is it possible to measure the benefits of reforms if no crises have occurred?
Wouldn't the lack of a counterfactual render an assessment of the benefits of reforms
impossible? These questions are not new, and they are in fact similar to cost-benefit analysis in
other policy areas. It is in the very nature of counterfactuals that they cannot be observed. If
monetary policy keeps the inflation rate at or below a target of, say, 2 %, how does one quantify
the macroeconomic benefits of having avoided an acceleration of inflation? If educational policies
increase school enrolment rates, how does one quantify the benefits of reduced youth
unemployment and perhaps even reduced youth crime rates?
The need to quantify benefits of (economic) policies without knowing the counterfactual is not
unique to financial sector reforms. There are two - complementary - approaches of dealing with
them:
First, assessing the benefits of reforms can be done without directly measuring the broad
objectives of reforms in terms of enhanced stability of (global) financial markets or a lower
probability of financial crises. Indeed, the benefits of reforms can be assessed by looking at the
channels through which excessive risk-taking in the financial sector materializes and through
which risks to financial stability can build up. The focus can be on indicators that are closely
related to factors contributing to financial sector stress. In the same vein, indicators might include
the soundness of the financial sector and the contribution of the financial sector to real economic
activity.
This then leads to the second question, namely the question of establishing a causal impact of
reforms on these indicators and the construction of counterfactuals: How would the financial
system have developed without the reforms having been implemented?
Constructing macroeconomic counterfactuals is difficult and not desirable. We do not wish to
see a financial crisis in a country that has not undertaken financial sector reforms just so we can
improve our understanding of the benefits of reforms in a country that has undertaken such
reforms. So, while the "real" counterfactual is neither possible nor desirable, there are
nevertheless many ways to construct synthetic counterfactuals that can serve a similar purpose
and many ways to use econometric methods that make it possible to establish causality. There
are many possibilities of dealing with the challenge of causal identification, even in the area of
macroeconomics (Fuchs-Schündeln and Hassan 2015). But empirical methods alone are not
enough; good conceptual and theoretical frameworks are also needed to decide what to look for
in the data.
b) Have reforms had similar effects across markets and jurisdictions (heterogeneity)?
Reform effects differ across countries, banks, and sectors of the financial services industry. This
heterogeneity facilitates rather than prevents reform evaluation. Essentially, heterogeneity and
institutional knowledge are important for good (causal) impact assessments. Comparing
countries that enacted different types of reforms at different points in time, comparing
developments before and after reforms have been initiated, comparing banks for which new
regulations are more or less binding can help to establish causal effects. Understanding
differential effects across markets or jurisdictions delivers valuable lessons for policy
development that would be missed by looking only at averages.
Take, for example, the increase in capital requirements for banks after the crisis. Higher capital
requirements increase the resilience of banks with regard to shocks and provide the basis for
sustainable lending to the real economy. Trends in post-crisis bank lending have differed across
countries and individual banks. A decline in lending can, for instance, be due to a decline in credit
demand, more prudent lending behavior of banks in response to higher risks, or it can result from
a decline in banks' credit supply. Such a decline in credit supply, in turn, can be caused by
structural changes in the banking sector and the reallocation of market shares to non-bank
lenders, it can be due to a re-orientation of banks' internal strategies, or it can be the
consequence of tighter regulations. Experience shows that all of these factors play a role and
that their relative importance differs.
What conclusions can be drawn from observing trends in bank lending over time? Is direct
inference possible with regard to the intended and unintended consequences of reforms? Prima
facie, aggregate time trends and averages provide only limited answers to these questions. In the
extreme, basing policy decisions on such statistics alone can aggravate the situation. For
example, regulations may be relaxed because of their alleged impact on credit supply, even if
they did not play a major role in the contraction of credit in the first place. Banks may then be
encouraged to lend excessively in risky markets - with potentially negative effects for financial
stability.
c) Has the reform achieved its overall objective (general equilibrium)?
In order to assess the effects of reforms, it is ultimately necessary to conduct a general
equilibrium analysis. Which markets are affected, what are the interactions between different
reforms, how does the allocation of risk and credit change in an economy? These are questions
which cannot be answered directly by microeconomic models that carefully identify causal
effects.
At the same time, general equilibrium analysis is even more challenging than partial equilibrium
modelling, in particular in situations in which markets dynamically adjust and many shocks hit the
system simultaneously. Most of the time, financial markets are not "in equilibrium" but in a state
of transition.
Rather than promising to deliver a comprehensive view of all market adjustments, general
equilibrium analysis serves the purpose of identifying the relevant shocks, transmission
channels, and market interactions. Rather than broadening the perspective even further, a
general equilibrium perspective should help to focus on the relevant issues under consideration.
IV. A joint task for academics and policymakers
Academics and policymakers can join forces to conduct good policy evaluations. Post crisis, the
academic profession and economists in particular have been criticized for their inability to predict
3
crises, for using the wrong models, and for applying the wrong concepts. Large parts of this
criticism ignore the significant progress that has been made with regard to economic analysis in
performing meaningful causal evaluations, modeling heterogeneity, and exploiting
microeconomic data. Enhancing the use of existing data, methodologies, and the skilled
personnel that are employed both inside and outside policy institutions should thus be a priority.
Both academics and policymakers would benefit from engaging in better policy evaluations.
Academia could engage in developing methodologies and in studying designs that contribute to
societal welfare without compromising on academic rigor. Policymakers could draw on the rich
expertise that is available and make better use of existing infrastructures. Moreover, some of the
mechanisms that have been developed in academia to mitigate misaligned incentives and
ensure transparency are readily applicable to policy evaluations.
Good policy evaluation needs to take incentives of the different stakeholders into account. For
example, regulators have an incentive to overstate the "success" of reforms, industry has an
incentive to focus on private costs, while academia has an incentive to focus on "novel" aspects.
Policy institutions may lack resources to govern the process. Incentives can be managed
according to the following three basic principles:
1. Transparency
Transparency is a precondition for a credible regulatory process. Goals and concepts need to be
well-defined. Policy evaluation means being transparent about the goals of regulatory policies
and what these policies have actually achieved.
Transparency can be accomplished by making information on evaluation studies, data (data
confidentiality permitting), and codes available to relevant stakeholders. A centralized collection
of evaluation studies can reduce the costs of obtaining relevant information for everybody. Such
a repository can improve transparency by making it easier and less costly for (internal)
evaluators and external stakeholder to keep track of evaluation work. Technical solutions can
range from a website to more structured repositories. Other fields, such as development
4
economics or medicine often rely on repositories in order to structure available evidence.
Replicability is an important element of transparency. Replicability ensures that policy decisions
are based on robust evidence and on results which survive changes in the set-up of the
empirical model. This requires transparency about the models being used, access to data so
that results can be replicated, and good documentation of the set-up of studies. Many of the
required protocols have been developed in academia and can be applied to policy work. At the
same time, policy evaluation would benefit from a greater appreciation of replication studies in
academic work.
In order to ensure replicability, democratic data access is important. Robust evaluations have to
rely on a wide range of methods and good data. Whenever possible, data used in the analysis
should be made public. This facilitates the replication of results and can, thereby, help to ensure
high quality standards.
2. International coordination
Regulatory reforms have a global dimension and have been decided at an international level.
5
Also, prudential policies have international spillover effects. Speaking a common language is
thus important when looking at the effects of reforms. Setting standards, learning from good
practices, and international coordination are vital. The FSB, in coordination with international
standard setting bodies, can play an important role in this regard. Its proposal for a framework for
policy evaluation comes at the right time and addresses the right issues.
3. Independence
Independent, objective assessments are needed to obtain an unbiased picture on the effects of
reforms. There are several avenues towards independence. External experts might be involved
that have a mandate for independent evaluations and do not have a stake in the formulation of
policies. These can be existing institutions such as research institutes or policy advisory bodies.
Internal experts might also provide independent assessments, in particular in cases where data
cannot be shared with externals. Internal evaluations, however, are at greater risk of being
subject to implicit or explicit biases. Therefore, appropriate institutional arrangements such as
peer reviews, independence from policy groups, and direct reporting lines need to be in place.
V. References
Basel Committee on Banking Supervision (2010), An assessment of the long-term
economic impact of stronger capital and liquidity requirements, Bank for International
Settlements, August 2010.
Buch, Claudia M., and Oliver Holtemöller (2014), Do We Need New Modelling Approaches in
Macroeconomics?, IWH Discussion Papers 8/2014.
Buch, Claudia M., Matthieau Bussiere, and Linda Goldberg (2017). Prudential policies
crossing borders: Evidence from the International Banking Research Network. Vox EU.
voxeu.org/article/prudential-policies-crossing-borders
Financial Stability Board (2016). 2nd Annual Report on the Implementation and Effects of the
G20 Financial Regulatory Reforms. August. Basel.
Financial Stability Board (2017). Proposed Framework for Post-Implementation Evaluation of
the Effects of the G20 Financial Regulatory Reforms. April. Basel.
Fuchs-Schündeln, Nicola, and Tarek Hassan (2016). Natural Experiments in
Macroeconomics, in: Taylor, John B., and Harald Uhlig (eds.): Handbook of
Macroeconomics, Elsevier, Vol. 2a: 923-1012.
Laeven, Luc, and Fabián Valencia (2012). Systemic Banking Crisis: An Update. IMF Working
Paper 12/163, Washington DC.
Rey, Helene (2013). Dilemma not Trilemma: The global financial cycle and monetary policy
independence.
voxeu.org/article/dilemma-not-trilemma-global-financial-cycle-and-monetarypolicy-independence
Verein für Socialpolitik (2015). Leitlinien und Empfehlungen des Vereins für Socialpolitik für
Ex post Wirkungsanalysen. September.
www.socialpolitik.de/docs/VfS-Leitlinien_Ex_postWirkungsanalysen.pdf
Wissenschaftlicher Beirat beim Bundesministerium für Wirtschaft und Energie (2013).
Evaluierung wirtschaftspolitischer Fördermaßnahmen als Element einer evidenzbasierten
Wirtschaftspolitik. Berlin.
1
See, e.g., Basel Committee on Banking Supervision (2010).
2
an evaluation framework applicable to a range of regulatory policies in the European Union,
sFor e e ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htmec.europa.eu/smart-
regulation/guidelines/toc_guide_en.htmec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm. The
German Economic Association (Verein für Socialpolitik 2015) and the academic advisory council to the German
Ministry of Economics (Wissenschaftlicher Beirat 2013) have likewise developed criteria for good,
evidencebased policymaking.
3
For a review of this discussion, see Buch and Holtemöller (2014).
4
Examples include the Health Systems Evidence from the McMaster Health Forum
(www.healthsystemsevidence.org), the Cochrane Library (www.cochranelibrary.com), or, for development
economics, J-PAL (www.povertyactionlab.org/) or the International Initiative for Impact Evaluation
(www.3ieimpact.org).
5
Buch, Busssiere, and Goldberg (2017) summarize the results of a cross-country research project on prudential
spillovers through international bank lending. Rey (2013) discusses the importance of prudential policy for
global financial cycles.