Claudia Buch: Evidence-based policy - potentials, infrastructures
and the role of international organizations
Speech by Prof Claudia Buch, Vice-President of the Deutsche Bundesbank, at the School of
International Studies (ZIS), Dresden, 24 May 2019.
* * *
The paper has been prepared for the symposium on the occasion of the 20th anniversary of the
studies of international relations at the University of Dresden. It has benefited greatly from
comments and inputs by Edgar Vogel and Regina Riphahn. All errors and inconsistencies are
my own.
1 The Role of Evidence-Based Policy
Policy decisions in today's world have to be taken in a dichotomous state between good
evidence and public distrust in expert opinion.
On the one hand, the conditions for evidence-based policy have seen excellent progress.
Availability of and access to high-quality data have improved, academic research provides a
range of methodologies to conduct causal impact assessments, academic curricula incorporate
these methods, and administrations are staffed with highly qualified personnel. Overall, trust in
scientists appears to be strong. Surveys for the US show that trust in scientists has been at a
1
fairly high and stable level since the 1970s . Surveys for the US and the UK reveal areas where
2
the public trusts scientific experts more than the government . Most importantly, trust is not only
inspired by the perceived quality of policy outcomes: rather, citizens want a transparent and
3
objective decision-making process.
On the other hand, trust in governments and political institutions in particular seems to have
eroded since the Global Financial Crisis. Many public policy debates are characterized by
4
skepticism via-a-vis experts and expert knowledge. Populism is on the rise in many countries.
83% of the respondents to a Eurobarometer poll see "fake news" and online disinformation as a
5
threat to democracy.
Against this background, evidence-based policy is an undogmatic concept. Rather than taking a
normative approach to policy, it adopts a positive perspective: Defining policy objectives remains
within the realm of the democratic policy process. But, given the policy objectives, better
evidence can contribute to selecting the right policy instruments, assessing the impact of these
instruments, and potentially, revising them.
The potential benefits of applying good analysis to problems facing today's societies are in fact
large. Historically, the interaction between scientific knowledge, policy-making, and
entrepreneurial activity catalysed by the Enlightenment has brought significant improvements in
6
human well-being. Notwithstanding the huge challenges facing us globally in terms of
inequalities, environmental risks, climate change, and global tensions, societal conditions have
improved across many dimensions. Improving information on which to base policy decisions
thus promises large potential gains.
However, putting evidence-based policy into practice is a challenging task and requires efforts
and contributions from several stakeholders. The remainder of this text presents examples of
evidence-based policy, discusses the role of international institutions in facilitating
evidencebased policy, and presents thoughts on how setting up the right infrastructures can reduce the
costs of evaluations and improve their quality.
2 Evidence-based policy can "work"
One way to promote an evidence-based agenda is to show that evidence-based policy in fact
works and that its challenges are common to many fields. Here are three examples from
financial regulation and supervision.
a) Evaluation of financial sector reforms
The global financial crisis has exposed the fault lines in the regulatory system for financial
markets. In 2011, G20 members thus agreed on reforms addressing the "systemic and moral
7
hazard risks associated with systemically important financial institutions". Policies have been
implemented in order to improve loss absorbency and resilience, recovery and resolution, and
supervision. These reforms target more narrowly defined objectives - by focusing on the
responses of individual banks - and more broadly defined objectives in terms of aggregate
outcomes such as the functioning of financial markets.
Many of these reforms have been implemented to date, and a first evaluation of their effects
becomes feasible. Such evaluations have to take a structured approach in order to balance an
assessment of costs and benefits, and to take a societal rather than private perspective. What
appears to be a cost of financial regulation to individual market participants may well be of benefit
to society overall by making the system more efficient and resilient.
As an instrument to operationalize such complex evaluation projects, the G20 leaders endorsed
a framework for the post-implementation evaluation of G20 financial regulatory reforms in
8
2017. The framework provides an umbrella for the Financial Stability Board's (FSB) policy
evaluation projects and serves as an orientation for the practical conduct of post-implementation
evaluations. It rests on two pillars. First, the framework contains provisions regarding the
decision-making process and the interaction of relevant stakeholders such as regulators or
market participants. This is an important element of transparency and accountability of the FSB.
Second, the framework provides guidance on analytical aspects of evaluations and engenders a
common understanding of what constitutes "good" evaluations and robust evidence.
Regarding analytical guidance, the framework stipulates that each FSB evaluation should answer
three key questions. First, did the reforms "cause" an outcome ("Attribution")? Second, did the
reform have similar effects across markets, states of the world, or jurisdictions and regions
("Heterogeneity")? Did the reform achieve its overall objective ("General equilibrium")? In the spirit
of the undogmatic concept outlined above, the answers to these questions should provide
detailed information to policymakers on the impact of their policies on observed economic
outcomes. They should enable informed policy discussions without pre-empting decisions
regarding adjustments of policies.
The first two evaluations under the framework were delivered to the G20-Summit in November
2018. The first project investigated to what extent post-crisis reforms incentivized central clearing
of OTC-derivatives. The second project evaluated the effect of reforms on the financing of
9
infrastructure. More projects are underway. An evaluation of reform effects on SME financing
will be delivered to the G20-Summit this year with a consultation report published soon - open to
comments from all stakeholders. Another project considering the FSB's reform agenda on
ending Too-Big-To-Fail (TBTF) has just started. This evaluation project will assess whether
reforms are effective in reducing the systemic and moral hazard risks associated with
systemically important banks. It will also examine the broader effects of the TBTF-reforms on the
overall functioning of the financial system. The final report is scheduled for the G20-Presidency in
10
2020.
b) Macroprudential policy
11
Macroprudential policy is a relatively new policy field. Its goal is to preserve financial stability
and to prevent the build-up of systemic risk that may have adverse effects for the functioning of
the financial system and for the real economy. New institutions have been tasked with the
implementation of macroprudential policies, and new policy instruments have been introduced.
Nonetheless, uncertainty about the state of the financial system and the effects and
effectiveness of these policy instruments is high. This uncertainty entails two risks: the risk of
acting too late (inaction bias) and the risk of choosing an inappropriate instrument or inadequate
calibration.
Both risks can be mitigated if macroprudential policy is embedded in a structured policy process.
Such a policy process involves four steps. In a first step, the policy objective(s) of
macroprudential policy need to be specified. Macroprudential authorities use different definitions
of the policy objective, but all aim at reducing systemic risk arising from externalities for the
functioning of the financial system. In a second step, intermediate objectives need to be
specified, and appropriate indicators need to be chosen. Intermediate objectives are linked to the
drivers of systemic risk such as leverage, risk-taking incentives, connectedness, or exposure to
common shocks. In a third step, the activation or recalibration of policy instruments that address
systemic risk externalities needs to be considered. The decision on whether and how to activate
policy measures should be based on a structured process of ex-ante policy evaluation. Such an
ex-ante evaluation provides information about the relative performance of different instruments in
contributing to reducing systemic risk. In a fourth step, and once sufficient time has elapsed, the
effects of the instruments need to be assessed in an ex-post evaluation. This step provides
information about the effectiveness of the measure(s) taken, about intended or unintended side
effects, and it also serves as an input into a possible recalibration of the policy instruments. In a
nutshell, this is what the FSB evaluations mentioned above is about.
The macroprudential regulation of residential real estate markets in Germany is a good example
12
for how the macroprudential policy cycle can be put into practice. Currently, macroprudential
instruments for the real estate market in Germany are not activated. Before activation of an
instrument, authorities are required to perform an in-depth ex-ante evaluation. The analysis
should quantify the expected impact, for instance on the resilience of the financial system or on
credit markets. The output of such an analysis should inform the calibration process and provide
an assessment of the expected costs and benefits. After activation of the instrument, a
compulsory ex-post evaluation should provide not only information on the instrument's
effectiveness but also on potential unintended consequences. The results from this step can
then feed into a potential re-calibration of the instrument.
c) Learning from other policy areas
With initiatives such as the one by the FSB, evidence-based policy has become part of the policy
process in the area of international financial regulation. There are other policy areas and
international initiatives which can inform this process - with regard to potential avenues to
success and pitfalls to be avoid. In order to promote learning from other policy areas and to
facilitate a structured dialogue between policymakers and academia, the German National
Academy of Science Leopoldina, in cooperation with the Deutsche Bundesbank, organized a
13
workshop in 2018. Here are examples of issues that were addressed in this workshop.
In Germany, the Federal Ministry of Finance has been conducting so-called "spending reviews"
since 2018; analysing revenues and spending decisions and scrutinizing their effectiveness and
14
efficiency. The German National Regulatory Control Council has developed a concept for
structuring ex-post impact assessments applying to all funds spend by the federal government
15
and, more recently, argued in the favor introducing quality standards for such evaluations.
An area in which policy evaluation in Germany has advanced most are labor market reforms. In
the early 2000s, the German government embarked on comprehensive labor market reforms
aimed at reducing unemployment, improving access to work, making new job relationships more
sustainable, and enhancing the efficiency of the employment agencies. Public interest in a
deeper investigation of the effects of these reforms was sparked by persistently high
unemployment rates and the perceived ineffectiveness of policy. There were a number of critical
factors that affected evaluations of these policies. First, the evaluation was designed with a view
to providing a systematic and broad overview. Second, academic research had a range of
appropriate tools available, making it possible to provide robust results within a reasonably short
time-span. Third, a mandatory reform evaluation was enshrined in law. Finally, a common
concept developed by participating research institutions provided guidance on practical and
analytical aspects of the evaluation.
Another example of evidence-based policy is the "What Works" Network, an initiative launched by
16
the British government in 2013. The network consists of independent What Works Centres
and affiliate members. The What Works Centre for local economic growth, for instance,
analyses which policies are most effective in supporting local economic growth. In practical
terms, it reviews and summarizes existing evidence in meta studies, builds databases on
relevant work or provides guidance to practitioners on how to meaningfully approach evaluations,
for example by providing technical training, supporting piloting and testing local economic
schemes or developing case studies which portray particularly useful evaluation techniques.
3 The role of international organizations
Evidence-based policy promises huge gains - better policies at potentially lower costs. But
evidenced-based policy is not a holy grail. It does not overturn the political dynamics that shape
policy discussions. However, it does promise a better, more structured, and more informed
policy debate.
At the national level, electoral cycles have a tendency to work against a structured evaluation of
policies. Good institutional design at the national level can thus serve to sustain policies and to
ensure bi-partisan support to evaluations. The advantage of entrusting independent institutions
with evaluations is that they can operate outside the perimeter of day-to-day pressure from
political debates. At the same time, the further away responsibility for policy evaluation is placed
from the political discourse, the more political accountability becomes an issue. One mechanism
that can be used to overcome this concern is enhanced transparency: public consultations,
transparency with regard to evaluation methods, replicability of studies, and clear responsibilities
are mechanisms to ensure transparency and thus credibility.
International organizations have an important role to play as well. They can enhance
transparency about evaluations and institutional designs. Rather than getting involved in detailed
discussions about national policy design - where national constituencies may question their
democratic legitimacy - international organizations can help setting up evaluation frameworks
and policy structures. This can be part of their surveillance work. For instance, the European
17
Commission's "Better Regulation Guidelines" provide guidelines on how the impact of EU-
regulations should be evaluated. The OECD has published a framework for regulatory policy
18
evaluation.
Obviously, objective cross-country evaluations are challenging, if not infeasible. Can we compare
the effects of policies that have been implemented with very different objectives and under very
different institutional settings? Despite these hurdles, benchmarking and improving information
on evaluations is a low-hanging fruit. International organizations provide a range of indicators.
However, information on the share of the underlying policy programs that are subject to
evaluation is not easily available. A survey run by the OECD shows that members have
progressed in making regulatory policy more fact-based. Ex-post evaluation has, however, not
19
always become second nature to institutions involved in the design of regulatory policies.
An additional important task for international organizations can be the establishment and
maintenance of repositories of evaluation studies. Repositories provide a compact overview of
relevant evaluation work in a certain field. Tailored keywords and meta-information allow
narrowing down the set of studies to be considered. In this way, repositories provide relevant
information for academics, policymakers, the public, journalists, and industry. This facilitates
more efficient evaluation work in the official sector (including international organizations), for
academic researchers, and the industry.
Repositories are well-established in some fields such as medicine, where the Cochrane Library
20
or the McMaster Health Forum provide structured evidence on medical research, also by
compiling meta-studies. Similarly, in the field of development economics, the International
Initiative for Impact Evaluation (3ie) collects information on the effects of policies and programs in
development economics. The J-PAL initiative does the same in the case of poverty-reduction
21
projects. The OECD's International Network on Financial Education (INFE) is a platform that
facilitates the exchange of experiences on policies promoting financial education. It does so by
collecting data and developing comparative reports. Finally, the Bank for International
Settlements (BIS) recently launched a "Financial Regulation Assessment: Meta Exercise
(FRAME)", which enhances the transparency of existing studies and improves efficiency in terms
22
of collecting relevant information.
4 Infrastructures supporting good evaluations
Many projects and initiatives show that policymaking can benefit from sound evaluations and
from making the best use of available knowledge. Evidence-based policy can help improve and
continue successful policies, while finding negative or unintended consequences can point to
areas for improvement. But evidence-based policy comes at a price. Rigorous evaluations take
time, and they require input of human resources and data. Not least, evaluations require good
communication on how to interpret findings and how to put them into perspective.
These costs of evaluations can be reduced by setting up the right infrastructures:
Legal mandates and administrative procedures: There will be a natural tendency of policy
evaluations to be in a time and resource conflict with other administrative tasks. Moreover,
unless clear procedures for evaluations have been established, political considerations may
dominate the administrative agenda. Against this background, establishing and strengthening
legal mandates for mandatory evaluations can overcome political biases and the tendency
toward inaction. Clear legal mandates can help to identify and agree on policy objectives,
indicators, and benchmarks upfront. And they help to embed procedures for policy evaluations
into mandates and day-to-day work of administrations, thus helping to avoid conflicts with other
tasks.
Repositories and information platforms: Generating, using, and adapting evaluation projects will
happen more easily if relevant information is readily available. One way to facilitate the flow and
exchange of information are repositories of evaluation studies. In addition, an important
prerequisite for improving the dialogue between researchers and policymakers is that each side
should learn to "speak the language" of the other side, at least to some extent. This requires
clear communication of research results and methods, and the translation of high-level policy
objectives into measurable indicators.
Data infrastructures: Evaluations require good data, but data infrastructures can be costly. The
costs of acquiring data can be minimized if data are collected early on and, ideally, if data needs
are considered when drafting new legislation. Notwithstanding the key importance of ensuring
strict data confidentiality, broad use of the available data will help to improve the quality of
evaluations.
Managing incentives: Ultimately, policy evaluations will be used routinely only if the right
incentives for conducting them are in place. Administrations can be incentivized to conduct
evaluations through institutional structures which are conducive to evaluations and legal
mandates to evaluate policies. The incentives of researchers will depend, ultimately, on the
research community's criteria for assessing the quality of academic work. Giving researchers
access to data can be an incentive to cooperate. But the academic community can do more:
rewarding replication studies can provide additional incentives to cooperate.
Bringing more and better evidence into political decisions is beneficial to society. To make
progress, we need commitment from and an open and continuous dialogue with all stakeholders.
The more we can build on available infrastructures, the lower the costs of evaluations will be, and
the better we can learn from past experience. Good policy design and good institutions may,
ultimately, increase transparency and accountability, thus countering skepticism about "expert"
projects.
1
Funk, Cary (2017), "Real Numbers: Mixed Messages about Public Trust in Science Issues in Science and
Technology", Volume XXXIV Issue 1, Fall 2017.
2
Osman, Magda, Norman Fenton, Toby Pilditch, David Lagnado, and Martin Neil (2018), "Who do we trust on
social policy interventions", Basic and Applied Social Psychology, Vol 40(5), p. 249-268.
3
OECD (2017), "Trust and Public Policy: How Better Governance Can Help Rebuild Public Trust", OECD Public
Governance Reviews, OECD Publishing, March 2017.
4
Quoting Conservative MP Michael Gove ahead of the Brexit-referendum: "The people of this country have had
enough of experts".
5
ec.europa.eu/digital-single-market/en/news/final-results-eurobarometer-fake-news-and-online-disinformation
6
Pinker, Steven (2017), "Enlightenment Now: The Case for Reason, Science, Humanism, and Progress";
Rosling, Hans, Ola Rosling, and Anna Rosling Rönnl (2018), "Factfulness: Ten Reasons We're Wrong About
The World - And Why Things Are Better Than You Think".
7
See Financial Stability Board (2011), "Policy measures to address systemically important financial institutions",
November 2011.
8
See Financial Stability Board (2017), "Framework for Post-Implementation Evaluation of the Effects of the G20
Financial Regulatory Reforms", July 2017.
9
Final documents on both projects are available on the FSB's website.
10
Financial Stability Board (2018), "FSB Chair's letter to G20 Finance Ministers and Central Bank Governors",
March 2018.
11
This section is largely based on Buch, M. Claudia, Edgar Vogel, and Benjamin Weigert (2018), "Evaluating
macroprudential policies", European Systemic Risk Board (ESRB) Working Paper 76.
12
The legal basis is §48u of the German Banking Law (Kreditwesengesetz) as published in Bundesgesetzblatt
(2017), Nr. 34, 9. June 2017.
13
The findings from this workshop have been summarized in Buch, Claudia M. and Regina T. Riphahn (2019),
„Evaluierung von Finanzmarktreformen - Lehren aus den Politikfeldern Arbeitsmarkt, Gesundheit und Familie",
Leopoldina Forum Nr. 1.
14
www.bundesfinanzministerium.de/Content/EN/Standardartikel/Topics/Public-Finances/Articles/2017-06-14spending-reviews-im-bundeshaushalt.html
15
Nationaler Normenkontrollrat (2018), "Germany: Less Bureaucracy, More Digital Services, Better Regulation.
Let's get to it!", Annual Report of the National Regulatory Control Council 2018, October 2018.
16
www.gov.uk/guidance/what-works-network
17
ec.europa.eu/info/sites/info/files/better-regulation-guidelines.pdf
See OECD (2014), "OECD Framework for Regulatory Policy Evaluation", OECD Publishing, June 2014. The
framework is a natural follow-up to the Recommendation of the Council of the OECD on Regulatory Policy and
Governance from 2012.
19
See OECD (2015), "OECD Regulatory Policy Outlook 2015", OECD Publishing, May 2015.
20
www.mcmasterforum.org/
21
www.financial-education.org/join_INFE.html
22
See stats.bis.org/frame/ and Boissay, Cantú, Claessens, and Villegas (2019)," Impact of financial regulations:
insights from an online repository of studies", BIS Quarterly Review, March 2019.