My Vision No.45
2019.12

Does scientific analysis enhance
the quality of policy?
Seeking to realize more reliable administration, the Japanese Cabinet decided to promote theintroduction of Evidence-Based Policymaking (EBPM) in 2017. Since then, a system for thepromotion of the method has been created, proving trials have been conducted in ministries andagencies, and capacity building for staff members is proceeding.
In order to be certain regarding the improvement of the quality of policy, it will be necessary tounderstand the characteristics of EBPM – a method originating in the field of economics – identify
its significance and its limitations, and ensure that it appropriately takes root in Japanese society.
Now is precisely the time to discuss what we need to do in order to make EBPM effective.

About this Issue
What constitutes evidence?
– Demonstrating the causal relationship between measures and outcomes
Noriyuki Yanagawa

NIRA Executive Vice President, Professor, Graduate School of Economics, The University of Tokyo
Keywords…Scientific basis, EBPM, consideration of causal relationships, appropriate understandingof randomized controlled trials, political factors, how to collect evidence, public understanding of
scientific policymaking

Expert Opinions
Does scientific analysis enhance the quality of policy?
What is the significance of EBPM? What are its limitations?
What are the issues that we face in attempting to advance the method?

Design incentives for the use of evidence

Link the internal and external resources of
the government

Daiji Kawaguchi

Yohei Kobayashi

Professor, Graduate School of Economics,
The University of Tokyo
Keywords…Influence of evidence in policymaking,independent analysis by statistics departments
within ministries and agencies

Chief Analyst, Mitsubishi UFJ Research andConsulting Co., Ltd.
Keywords…Causal relationship between measuresand outcomes, basis for causal relationships,
potential for reallocation of resources, randomized
controlled trials (RCTs)

Excessive confidence can lead tomisinterpretations

The limitations of "Evidence-based
Policymaking"

Makiko Nakamuro

Martyn Hammersley

Professor, Faculty of Policy Management,
Keio University
Keywords…Correlation, causal relationship,“Scared Straight,” rigorous policy evaluation

Both the public and private sectors must
become data-driven
Tsutomu Watanabe

Professor, Graduate School of Economics,
The University of Tokyo
Keywords…Utilization of private sector data andexpertise, data-driven policymaking, sharing ofroles between public and private sectors

Emeritus Professor of Social and Educational
Research, Faculty of Wellbeing, Education &Language Studies, The Open University
Keywords…Political defensibility, practicalexperience, practical judgment, context of policy
implementation, qualitative methods

Interview period：September, 2019
Interviewer: Kozue Sekijima(NIRA Research Coordinator, Researcher）

This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12
About this Issue

What constitutes Evidence?
– Demonstrating the Causal Relationship between Measures and Outcomes
When making policy decisions, it is important to
conduct discussions based on scientific evidence rather thanmere impressions and assumptions. As Professor Daiji
Kawaguchi of The University of Tokyo emphasizes in thisissue of My Vision, ours is an era in which the necessity forestablishing an order of priority in budget allocation isincreasing, and this in turn makes it necessary to establish
mechanisms for the formulation of policy on the basis of solid
evidence. Evidence-based policymaking (EBPM) is based on
Noriyuki Yanagawa
precisely this idea, and its importance has come to be
NIRA Executive Vice President,
Professor, Graduate School ofrecognized in recent years. However, in order to realize
Economics, The University ofpolicymaking of this type, it is also essential to examine howTokyo
to collect and analyze solid evidence, and how to utilize that
evidence in policy formation. In this issue of My Vision, experts in the field discuss these points
from a variety of perspectives.

Numbers do not equal evidence
Professor Kawaguchi also points out the importance of government agencies making
greater use of data and creating the conditions for evidence to actively influence policymaking. He
indicates that in the absence of active efforts in this area, even if ministries and agencies formulate
statistics and prepare data, little progress will be made. This is an important point against thebackground of the occurrence of problems in government statistics, the implementation of reviews,
and discussion concerning the best direction for the future.
But it is not enough to simply conduct discussions using numbers. It is a common patternthat numbers and data that suit one’s own preconceptions are found and presented as evidence –but such discussions cannot be considered to be solidly evidence-based. This point is emphasizedin this issue by Dr. Yohei Kobayashi of Mitsubishi UFJ Research & Consulting and Professor
Makiko Nakamuro of Keio University. Dr. Kobayashi indicates that EBPM is not simply a matterof using numbers in the discussion of policy, defining the method instead as “an effort to selectbetter policy measures while examining the causal relationship between measures and outcomes.”
Lack of appropriate consideration of causal relationships is a major mistake that is often
observed in the utilization of data. In this issue, Professor Nakamuro explains the difference
between relationships of correlation and relationships of causation using an easily understandable
example. However, in reality, it is rather difficult to verify causal relationships. This makes it
necessary to pursue the method of “randomized controlled trials,” or RCTs, as specified by both Dr.
Kobayashi and Professor Nakamuro. RCTs are a method that makes it possible to rigorously
define causal relationships, something that has attracted renewed attention with the winning ofThis is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12
this year’s Nobel Prize in Economic Sciences by Abhijit Banerjee, Esther Duflo and Michael
Kremer as a result of research using the method.
As Dr. Kobayashi tells us, because the introduction of EBPM in Japan commenced withdiscussion of reform of the method of formulating statistics, Japanese EBPM has tended to focus
on the preparation of data, but the method is inherently one that verifies causal relationships inrelation to policy measures. Professor Nakamuro also indicates the importance of the correct
identification of causal relationships, telling us that there are cases in which the failure toaccurately identify causal relationships leads to adverse effects.

Political factors are also importantEmeritus Professor Martin Hammersley of the UK’s Open University points out to us
that RCTs are not the be-all and end-all in policy-making. Indicating that RCTs are frequently
only able to demonstrate the causal relationships assumed by the researchers, while othervariables might actually be in play, and insisting on the importance of political factors such as
the inclinations of the implementers of the policy and the people affected by the policy, Professor
Hammersley warns that it is not possible to base policy directly on the results of experiments. It
is clear that while RCTs enable us to rigorously demonstrate causal relationships, this does notmean that policy can be formulated directly on this basis. This is a point that needs to beadequately understood.
No matter what the case, the key to enabling rigorous policy discussion based onevidence will be whether or not it is possible to collect a sufficient amount of evidence. In Japan,
there is still a dearth of evidence that can be used effectively in the formulation of policy, andhow to collect this evidence will be the major issue. In relation to this point, Professor Tsutomu
Watanabe of The University of Tokyo indicates the importance of making use of the private
sector. Clearly, this is the age of data, and data is considered to be a goldmine; it is an age inwhich the private sector is consequently expending considerable effort on accumulating a wide
variety of data. Given this, it will be necessary to give up the concept that the government mustcollect all of the data that forms the basis of policy, and embrace the concept of using private
sector data. Professor Watanabe also suggests that we should not only use private sector data,
but also make use of the power of the private sector for data analysis and evaluation.
The contemporary era is one in which governments no longer act alone in many areas,
but rather actively make use of public-private partnerships and the creative ingenuity of theprivate sector. EBPM is another area in which we should make greater use of the intelligence andabilities of the private sector. In attempting to determine policy in a more scientific fashion, it will
also be necessary to promote greater understanding among the public as a whole as to what
constitutes evidence and what significance evidence-based policy decisions will have for policy
formation. It is my hope that this edition of My Vision will help in this.

Professor Yanagawa is NIRA’s Executive Vice President, and a Professor in the Graduate School of Economics of The
University of Tokyo. He took his Ph.D. in Economics from The University of Tokyo. His research specializations arecontract theory and financial contracts.

This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12Expert Opinions

Design Incentives for the Use of Evidence
Social security expenditure in Japan is increasing, and thebudget that can be freely allocated by the government is declining.
Against the background of an increasing necessity for theestablishment of priorities in budget allocation, EBPM possesses
considerable significance in helping us to identify effective policies
based on evidence and focus resources on those policies as amatter of priority. In order to make EBPM effective, it will beessential to create mechanisms allowing policy formation on thebasis of evidence. Nevertheless, it can also be seen that EBPM –Daiji Kawaguchi
the concept of policymaking based on statistical analysis – is notProfessor, Graduate Schoolfunctioning in practice in Japan. Improvements must be made inof Economics,
order to enable evidence to have a greater impact on policymaking.
The University of Tokyo
For example, in the tripartite councils (government, laborand employers) that decide policy in the Ministry of Health, Labour and Welfare, the ministry
maintains a neutral stance as the Secretariat and attempts to find points of common groundbetween the respective claims of the labor side and the employers’ groups. The traditional cultureof the Ministry is to consider the consistency of these points of common ground with the law, andto formulate policy on this basis. I believe that the effectiveness of policy has been maintained
through the formulation of policies that satisfy the three parties to the process, but thispolicymaking mechanism tends to have a low level of regard for evidence. A survey is made of thestatus of wage revisions in order to provide evidence when determining the degree of increase in
the minimum wage. The survey method is decided on by the council that determines the minimum
wage, but the issue is that the evidence is neglected in decisions on policy and changes in theindustrial structure are not reflected. The problem is that the generation of evidence issubordinated to the decision on policy.
It will be necessary to make the transition to a system in which the departments within
Japan’s government ministries and agencies responsible for the generation and analysis ofstatistics conduct independent analyses and compile evidence, and that evidence is reflected that
evidence in the policymaking process. The utilization of evidence in policy decisions would provide
an incentive for the further promotion of EBPM, and attract superior human resources toministerial departments that oversee statistical analysis. If such an incentive did exist, it would
be necessary to give serious consideration to how to make use of administrative data while
protecting personal information. If we do not design incentives for the use of evidence, even if we
cultivate human resources and prepare data, efforts will simply end without bearing fruit. The
fact that despite the enormous amount of administrative data that have been collected, their use
as statistics has not advanced, is the result of systems that make it difficult to provide incentives
for the use of these data by the responsible ministries and agencies.
Professor Kawaguchi specializes in labor economics and empirical microeconomics. He holds a Ph.D. in economics from
Michigan State University. He took his present position in 2016 after holding positions including a professorship in
Hitotsubashi University’s Graduate School of Economics. He served as the Deputy Director of The University of Tokyo’sCenter for Research and Education in Program Evaluation (CREPE), and has been the Director of the institution sinceApril 2019. The purpose of CREPE is to develop the necessary methods and cultivate the necessary human resources forthe design and evaluation of policies and systems to be put into effect by ministries and agencies, local governments, andcompanies.

This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12Expert Opinions

Link the Internal and External Resourcesof the Government
EBPM represents an effort to select better policy measureswhile examining the causal relationship between measures andoutcomes. For example, let us assume that the government isconducting a remedial education program to help children in poor
households. In this case, the outcomes consist of academic
achievement and enrollment rates; the basis for the causal
relationship that indicates whether or not the program has
improved the outcomes is evidence, and it is EBPM that allows us
to refer to evidence in order to select better measures. In many
Yohei Kobayashi
cases, there are a number of potential measures that might be
Chief Analyst, Mitsubishi
implemented in order to achieve specific outcomes. The
UFJ Research andsignificance of EBPM is that it enables us to clarify the causal
Consulting Co., Ltd.
effects of the measures, and to redistribute resources such as taxes
and human resources to cost-effective measures that can be expected to produce outcomes.
One method of identifying the causal effects of policy measures is to compare outcomes forgroups for which the policy has been implemented with those for groups for which the policy has
not been implemented. Here, the important thing is how to assign the groups. In order to evaluate
the outcomes of the remedial education program mentioned above, it would be necessary toeliminate factors other than the measures implemented, such as the degree of parental
enthusiasm for education and the children’s own degree of motivation. In the case of a policy thatprovided such a program to families who apply directly, there is a risk that there may, for example,
be a bias in the subjects of the policy towards households that are passionate about education. In
such cases, “randomized controlled trials” are a very useful tool. This method randomly assigns
those to whom the policy is applied and those to whom it is not, in order to prevent a bias towards
specific attributes of the subjects, making it possible to identify the causal effects of the policy by
calculating the difference in outcomes between the two groups.
Because Japan’s initiatives in the area of EBPM commenced with improving the accuracyof statistics, there is a tendency towards the mistaken view that this is equivalent to EBPM. The
essence of EBPM, however, is making decisions while examining the causal effects of policy
measures. Elements such as the formulation of statistics, obtaining a grasp of the facts, andcreating logic models are important preparatory procedures for promoting EBPM, but in Japan
excessive effort tends to be focused on these areas. It will be necessary to return to the original
meaning of EBPM and examine the causal effects of policies.
There are limitations to the promotion of EBPM within the government alone. It would be
wise to link resources possessed by the government with external resources and accumulate
concrete case studies, beginning from small-scale and simple cases.
Dr. Kobayashi specializes in public economics, econometric analysis, and evidence-based policymaking. He completed thedoctoral program at Hitotsubashi University’s Graduate School of Economics, taking a Ph.D. in economics. He took his
present position following terms in positions including Deputy Director of the Industrial Structure Policy Division of the
Ministry of Economy, Trade and Industry’s Economic and Industrial Policy Bureau. Dr. Kobayashi also servesconcurrently as a Consulting Fellow and Policy Advisor for The Research Institute of Economy, Trade, and Industry(RIETI), and provides explanations of EBPM for a variety of organizations, including Data StaRt, the local government
data utilization support site operated by the Statistics Bureau of the Ministry of Internal Affairs and Communications.

This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12Expert Opinions

Excessive Confidence can lead toMisinterpretations
Dai Tamesue, a former track and field athlete who has
represented Japan in three Olympics, once wrote an essay
entitled “A Leading Athlete’s Flat Feet.” He refers in this essay toa famous track and field athlete who is flatfooted. Hearing aboutsuch cases, many people would believe that people with flat feetare perhaps faster runners. However, as Tamesue indicates,
saying that people who run fast are flatfooted is different tosaying that if a person is flatfooted, that person is a fast runner.
He also adds that there is a tendency to develop superstitions insuch areas, relating the result for example to the specific
Makiko Nakamuro
characteristics of a prodigiously talented athlete or to the specificProfessor, Faculty of Policy
sequence of actions prior to the good outcome. In fact, it appears
Management, Keio Universitythat the majority of athletes taking part in the Olympics are notflatfooted.
This helps us to understand why EBPM is important. Just because people who run fast are
flatfooted (correlation) does not imply that because a person is flatfooted that person will be a fastrunner (causation). If excessive confidence is placed in cases for which only correlation has beenverified and measures are implemented throughout society on this basis, the expected outcomesmay not be achieved, or opposite effects may even occur. As an example, the lesson offered by the
documentary “Scared Straight” is a significant one. Filmed in 1978 and the recipient of an
Academy Award, Scared Straight showed a group of juvenile delinquents experiencing a threehour session with convicted criminals in jail. Following this, the same group was filmed again in
1980, 1987 and 1999. It was found that all of the individuals who had participated in the session
with the convicts had been rehabilitated, and there was no evidence that they had been involved
in serious crimes. Following this, the number of state administrations in the US adopting Scared
Straight as a program for the rehabilitation of juvenile delinquents increased.
However, researchers later conducted studies in which youths were randomly assigned toparticipate or not participate in a similar rehabilitation program, and their follow-up researchshowed that the youths who had participated in the program were more likely than those who had
not participated to commit crimes and be arrested. Policy evaluations of this type are called“randomized controlled trials.” Such trials are well known as methods of determining whether arelationship is causal rather than correlational. It is desirable for EBPM to adopt rigorous methodsof this type in order to clarify the nature of causal relationships. Young people of school age change
quickly in terms of development and emotional state, and in such cases it is an easy matter tomistake changes that would have occurred even if the policy had not been implemented foroutcomes of the policy. Rigorous policy evaluation is necessary in order to prevent theimplementation of the policy from having the opposite effect to the intended effect, as in the caseof Scared Straight.
（Submitted manuscript）
Professor Nakamuro specializes in the economics of education economics, a discipline that employs economic theory andtechniques in the analysis of education. She identifies the effects of education policy in joint research with local
governments and schools. She received her doctorate from Columbia University, and took her present position following
periods of employment with the Bank of Japan and the World Bank. Professor Nakamuro also serves as a member ofgovernment councils including the Industrial Structure Council and the Regulatory Reform Promotion Council.
This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12Expert Opinions

The Limitations of "Evidence-based
Policymaking"

Martyn Hammersley
Emeritus Professor of Social and
Educational Research
Faculty of Wellbeing, Education &Language Studies, The Open University

It is sometimes believed that Evidence-Based
Policymaking(EBPM) will produce a dramatic
improvement in the quality of public policymaking. This
expectation is unrealistic. Research evidence cancertainly improve policymaking, but cannot guarantee todo this.
Public policymaking should, of course, be informedby evidence. But the concept of EBPM restricts evidence
to research evidence, and often to that from systematic
reviews of randomised controlled trials (RCTs).
Furthermore, this evidence is taken to show ‘what works’,and what does not; and thereby to tell policymakers what
to do. There are several problems with this:

1. It involves misleading assumptions about the policymaking process. Likely effectivenessof policies is not the only relevant consideration that policymakers must take into account.
Also relevant, for example, are whether a policy will be politically defensible; whether it
entails significant costs compared to alternatives; and whether it will have undesirable
longer-term effects on the attitudes and behaviour of those who must implement or
respond to it.
2. It underestimates the value of practical experience, and the need for practical judgment
in policymaking.
3. It overestimates the likely validity of evidence from RCTs. While certainly a powerfulresearch method, an RCT cannot control all potential threats to validity. For instance, indrug trials it is often possible to standardise the treatment, and measure outcomesaccurately, but these goals are much more difficult, if not impossible, to achieve in testingsocial and educational policies. Also, RCTs provide little information about contextual
variation in the effectiveness of policies; but this is crucial because the implementation
context of a policy will often be significantly different from the context(s) in which theevidence was produced.
4. It underestimates the value and validity of evidence produced by other kinds of research,
including that using qualitative methods. This can provide in-depth understanding of theassumptions on which a policy is based, some of which may be open to question; and it
can offer essential knowledge about the perspectives and practices of those who are to
implement the policy, and those affected by it.
Over time, the concept of EBPM has been liberalised, for example by requiring only thatpolicy be informed by, rather than based on, evidence, and by broadening the kinds of evidence
deemed acceptable. This liberalisation is beneficial: in particular, it acknowledges the limits to therole that research evidence can play in policymaking. However, it undercuts the novelty of EBPM,
and many of the claims often made on its behalf.
（Submitted manuscript）
A British sociologist, Professor Martyn Hammersley taught at the Open University from 1975 until his retirement in2015. He has carried out empirical research in the sociology of education, and on the reporting of research findings in themass media. However, much of his work has concerned research methodology in the social sciences. One of the areas towhich he has given attention is the role of research evidence in policy formation. Professor Hammersley is the author ofnumerous books.
This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement

My Vision No.45
2019.12Expert Opinions

Both the Public and Private Sectors
must become Data-driven
The Japanese government decided on a policy of promoting
EBPM at a Cabinet meeting in 2017, and we are now at the stage at
which each of the nation’s ministries and agencies has commenced
putting EBPM into practice. The establishment of systems is
progressing, but EBPM has not yet penetrated to the site ofpolicymaking. The main reason for this is that the government is not
making use of the power of the private sector. This has two meanings.
One of these is that the government is not utilizing the data that theprivate sector possesses. A diverse range of types of data is
Tsutomu Watanabe indispensable for scientific evaluations. Today, it is overwhelmingly
Professor, Graduate School
the private sector that possesses data. Unless the government isof Economics,
able to make more efficient use of private sector data, it is likely that
The University of Tokyo
the progress of EBPM will be slow. The other meaning is the factthat the government has little concept of leaving the analysis of data and the formulation of
statistics to the private sector in those areas in which the private sector is most capable of these
operations. The private sector is far in advance of the government not only from the perspective ofpossessing data, but also from the perspective of possessing the human resources required toanalyze those data. The government has no option but to make use of these resources.
The utilization of data has been slower in Japan than in other advanced nations. In the US,
the government and the central bank have begun to draw on the power of the private sector toenable data-driven policy formulation. The Federal Reserve Board operates a system by means ofwhich it obtains information from credit card processing companies, allowing it to rapidly gain an
accurate understanding of consumer trends. The FRB might analyze, for example, how exchange
rate changes affect the number of inbound tourists and the amount that each spends. This is aworld that traditional government statistics have not comprehended. There is a deft sharing ofroles with the private sector – for example, detailed and troublesome procedures such asprocessing the noise in raw data are entrusted to the private sector. This process, by means ofwhich both the public and private sectors make use of the data and expertise possessed by theprivate sector in order to accelerate data utilization, is also taking place in Europe.
In Japan, some financial institutions have begun to utilize data in this way, but at present
there is no sense of urgency with regard to the transition to data-driven decision-making in either
the public or private sectors. It is the universities that will be the key to the success of EBPM in
Japan. Universities have collective experience in the causal reasoning that is essential to EBPM,
and abundant human resources able to put it into practice. The creation of mechanisms enablingthese resources to be put to best use has already begun. It is to be hoped that this movement will
gain pace in the near future.
Professor Watanabe specializes in macroeconomics (prices and monetary policy), international finance, and corporate
finance. He is the leader of a large-scale project that uses micro price data to explicate the causes of deflation in Japan.
He obtained his Ph.D. from Harvard University, and took his present position in 2011, following terms at the Bank ofJapan and Hitotsubashi University. He was appointed dean of The University of Tokyo’s Graduate School of Economics
and Faculty of Economics in April 2019. In 2015, he founded Nowcast, Inc., a University of Tokyo venture company thatanalyzes big data in order to track movements in prices, consumption, etc. in real time for domestic and overseas financial
institutions and other organizations. Professor Watanabe is also the author of more than 70 papers focusing on monetary
policy and inflation dynamics.
This is a translation of a paper originally published in Japanese. NIRA bears full responsibility for the translation presented here.
Translated by Michael Faul.
Copyright Ⓒ 2019 by Nippon Institute for Research Advancement