Published in Technological Forecasting & Social Change. DOI information:
10.1016/j.techfore.2010.08.005. In Press.

PERCEPTIONS OF DOMAIN EXPERTS ON IMPACT OF FORESIGHT
ON POLICY-MAKING: THE CASE OF JAPANTatsuro Yoda
Institute for Future Technology
2-6-11 Fukagawa, Koto-ku, Tokyo 135-8473 JAPANt.yoda [at] iftech.or.jp

Keywords: Delphi method, impact, foresight, policy making, perception, survey

Abstract
In this study, the perceptions of domain experts who participated in foresight activities on theimpact on policy making are examined by conducting interviews and surveys on five previous
foresights in Japan between 1996 and 2007. The purpose of the study is to examine how
domain experts look at the practice of foresight in general, and perceive its overall impact onpolicy making in particular, in the setting of Japanese foresight conducted in the past 15 years.
There are two tasks for doing that in this study: to know the views of scientists and engineers
who participated in foresights on the impact of foresights they participated; and to know theeffect of respondent’s characteristics on their perceptions on impact.
There are two main findings of this study. First, the impact on policy making perceived bydomain experts who participated in foresight activities in Japan is not very high. Second, there
are different patterns of perception on the size of the impact on policy making depending onages, organizations, member status during foresight, and science and technology areas, andhabit of following the situation in general looking for any impacts after the foresight.

11 Introduction
An international group that analyzes methods for future technology analysis defines foresight as“a systematic process to identify future technology developments and their interactions withsociety and the environment for the purpose of guiding actions designed to produce a moredesirable future”[1]. Foresight techniques are defined as methods for conducting foresight, or,
methods aimed at “apprehending the longer-term future of science, technology, economy and/or
society in order to identify strategic research and emerging technologies coupled with economic
and social benefits” [2].
There are various kinds of evaluation studies on foresight activities such as studies on theaccuracy and reliability of its products [3; 4] its deliberation process [5; 6], its networking effects[7], its productivity [8], its regional differences[9; 10; 11; 12], its role at funding agencies [13], andits impact on policy making.
Martin and Johnston explain the benefit of foresight process as follows [7]:In conclusion, experience with foresight in the countries reviewed here suggests thatgovernment, industry, research and educational organizations, professional societies, andcommunity groups should all be encouraged to undertake, or to be involved in, foresight
exercises. Such exercises develop a better informed forum and a participatory andtransparent process for decision making on science and technology, allowing us toanticipate the potential consequences of current decisions. In short, technology foresightcan enable us to shape the future so that it better meets our longer-term economic andsocial needs.
In this process, “disparate group of people (academics, industrial researchers from different
sectors, policy makers, professional forecasters and scientific commentators)” are brought
together and “a structure within which they can communicate directly or indirectly with eachother” is made [14]. Communication is stimulated and partnerships are forged among
researchers, and between researchers, research users, and research funders [7]. Consensus isachieved among experts on an uncertain subject [15]. A feeling of commitment to the results offoresight is generated, the ideas generated are turned in action, and they are likely to be selffulfilling [14]. So the role of experts and their input into the process is very important for the
success of foresight activities and selection of experts is important for conducting high quality
foresight. The reliability of foresight depends on appropriate selection of experts [16].
Participation of well-respected experts is influential in persuading people that the foresight istaken seriously, and time and commitment are required from a range of experts to conduct
foresight successfully as a participatory process [17]. Keenan and Miles explain that experts’work include the following [17]:Gathering relevant information and knowledge;Stimulating new insights and creative views and strategies for the future;2



Helping to build new networks;

Diffusing the foresight process and results to much wider constituencies, and

Identifying and acting on ways to maximize the overall impact of foresight in terms offollow-up action

There are three types of participants in foresight: experts in foresight methods and organization,
experts in the domain(s) addressed by a particular exercise, and users of the outcome of theexercise [18]. The role of participants is important, and there are previous studies on foresightactivities including their processes and products from the standpoint of experts in foresightmethods and users of the outcomes such as government policy makers. In the explanation on
generational development of foresight in the past, it is pointed out that “domain expertise
becomes more significant” and experts have “never disappeared from the scene” [19]. But there
are not many previous studies on foresight activities from the standpoint of domain experts who
participated in foresights. One such study [20] found that “autonomy is still valued byresearchers and there was considerable antagonism towards many of the policies that impinged
on academics and were seen as an attack on them” based on analysis on UK foresight activities.
But is this statement valid to another foresight, for example, conducted in another country?
In this study, the perceptions of domain experts in foresight activities, whose types are varied interms of their science and technology areas or affiliated organization, on the impact of foresighton policy-making are examined by conducting interviews and surveys on participants in fiveprevious foresights in Japan conducted between 1996 and 2007. Japan initiated its first foresightusing Delphi method in 1972 and has been conducting foresight since then basically every 5
years [11]. In 2010, the ninth Delphi study was finished. The practice of foresight in Japan isconspicuous in terms of consistency of activities and involvement of a large number of scientists
and engineers.

1.1 Purpose of the study
The purpose of this study is, as stated above, to examine how domain experts look at thepractice of foresight in general, and perceive its overall impact on policy making in particular, inthe setting of Japanese foresight conducted in the past 15 years. There are two tasks for doingthat in this study. The first is to know the views of scientists and engineers on the impact offoresight activities they participated in Japan. What are their evaluations on impact? What kindof impact do they observe? On which actors and on which area of science do they perceive theimpact as high, or low? Are there any differences among the foresight activities perceived interms of the size of impact?
The second task is to know the relationship between the characteristics of the respondent andtheir perception on impact. There are various types of participation (committee member, Delphi
respondents, workshop participants, scenario writers, and so on), and various types ofparticipants (organization, scientist or engineer, science and technology area, age group, gender,3

distance from government, and so on). How are these differences in characteristics related totheir perceptions on impact?
By analyzing those, lessons, perspectives or recommendations are gleaned which was notfound by previous studies focusing on the views of method experts, and users of foresight.

1.2 Previous literature
In this section, previous literature on Japanese foresight is discussed. As stated above, foresightbased on the Delphi method started nearly 40 years ago in Japan, and has continued since then.
The foresights were evaluated by practitioners, academics, or users. One practitioner ofJapanese foresight explains its advantages as follows [21]:

The S & T community must periodically think seriously and in detail about the significant
science and technology trends relative to important socio-economic priorities and obstacles;Participation of science experts outside of the government helps maintain information flow
into the government and improves the ability to assess future demands on national
infrastructure; and

The Delphi provides a disciplined way to handle a broad range of topics, including new
and/or cross-cutting areas of science

He also emphasizes the role of foresight in building consensus, arguing that foresight provides“language” to communicate among Japanese actors in science, technology, and society. The
result of foresight is not only “sources of valuable general insight for policy makers andmanagers but also in some important cases have triggered action plans” [22]. He stresses thatthe strong interest in society in foresight was shown by the fact that around 3,000 copies of thereport were distributed in case of the fifth survey and major aspects were introduced to the
public by many newspapers and magazines [21].
Another practitioner in Japan also stresses that it is possible to collect wide range of S & T
information periodically by conducting Delphi survey. The collected information is comparable
across periods since the framework of survey is based on the framework used last time. From
the result of Delphi survey in Japan, it is possible to read the direction of technologicaldevelopment and agenda for promotion of research and development. He pointed out that thesignificance of the Delphi surveys is not on whether the predictions on realization oftechnological development is correct or not, but rather on contribution to promotion oftechnological development by showing the future direction of research and development. He
also emphasizes that the reliability of Delphi survey in Japan is enhanced by spending about
one year for selection of experts and topics for the Delphi survey. Selection of technologicalexperts and topics is important for conducing high quality Delphi survey. Only experts in eachtechnology domain who have a broad and balanced view on long-term science and technology

4trend can select appropriate topics, and the participation of knowledgeable experts in Delphi
survey makes the result of the survey reliable [16; 23].
The report of the National Institute of Science and Technology Policy (NISTEP), which is
responsible for foresights in Japan since the 5th Delphi survey in 19901, argues that there is“concrete influences (of foresight) on Japan’s research and development as well as technology
developments in general” [24]. In addition, NISTEP conducts a mail-based survey of private
firms that purchased the foresight report in 2000. The survey with 175 responses examined whopurchased the report for what purposes, how they utilized the results, and what kinds ofinformation they found useful. For about 75% of the respondents, the purpose of the usage ofthe report was to grasp the general trend of long-term science and technology development, andabout 60% thinks that the survey is valuable when making research strategy of each firm [25].
Irvine and Martin are the first scholars to pay attention to Japanese foresight in the early 1980s.
They emphasized that the main benefit is not the product of foresight but the process by which
forecasts are generated. Benefits of Japanese foresight summarized by them are basically thesame as the advantages stated above: [14; 26]
They provide a mechanism to ensure that researchers in all sectors, along with policymakers in government and industry, are periodically forced to think systematically about thelonger-term future;

The forecasts yield a general summary of what is happening, or likely to happen, across theentire range of R & D activities;



By surveying comprehensively the intentions and visions (and thus indirectly the current
strategic R & D activity) of the industrial research community, it provides a usefulmechanism for synthesizing major research trends across science-based sectors; andThe forecasts provide a useful mechanism for helping the government establish national
priorities in allocating resources

Cuhls pointed out that foresight in Japan brings in elements to moderate or negotiate between
the social interest groups and the results of foresight provide “the code to communicate betweensocial actors in science, technology, and society.” Earthquake research and solar cells arepointed out as two example of policy relevance of foresight in science and technology policy inJapan, since the fact that both topics were listed in topics for Delphi surveys contributed to thecontinuance of research in those areas [27]. Eto discusses that technology forecasting based onsuch consensus is considered in Japan to be reliable and the consensus promotes cooperation

1

The Science and Technology Agency was responsible for conducting the 1st to 4th Delphi
surveys. Since the 4th Delphi survey in 1986, the Institute for Future Technology has been
responsible for the operation including panel discussion and survey itself.
5and as a result the probability of success in research projects will be increased. According to him,
the wider the Delphi consensus is, the more accurate the Delphi forecast [28].
As to evaluation on reliability and accuracy, each foresight, conducted every 4 to 5 years,
analyzes the realization of results after 15 to 20 years. For example, in the 8th Delphi survey in
2004, the realization of the 1st (1971), the 2nd (1976), the 3rd (1981) and the 4th (1986) Delphi
results was examined by panel members. They found that 69% of 616 topics in 1st Delphi, 68%
of 641 topics in 2nd Delphi, 73% of 748 topics in the 3rd Delphi, and 66% of 933 topics in 4th
Delphi survey were realized or realized partly. Considering the fact that over 60% of the Delphi
topics are realized, it was pointed out that Delphi surveys have played the role of giving
scientists and engineers in Japan the targets of R & D, and thus promoting their realization [23].
As mentioned above, foresight in Japan has been evaluated as highly effective by previous
literature. It is a periodically conducted participatory process that covers broad areas of scienceand technology. It promotes communication and helps building consensus. The society and theindustry in Japan are interested in the results, which have been found relatively accurate. It
provides the government valuable information on science and technology. The government setsa priority based on the areas where the results of the Delphi survey show promising, andscientists and engineers make efforts around those areas.
But is this a real picture of what is going on? If this is real, to what extent? There are threequestions. First, there are many participants in the Delphi survey including both panel members
and respondents to Delphi survey as explained later. The Delphi survey sends a questionnaire
twice in order to narrow the divergence of views. Does this guarantee that the consensus isreached? Originally the purpose of conducting the Delphi survey was to seek consensus,
because consensus of expert opinion was hypothesized to be more accurate than an individual
forecast by an expert. But Delphi is now seen as “no more or less than a systematic means ofsynthesizing the judgments of experts – the aggregate judgment representing a kind ofcomposite expert composed, in the domain of interest, of the expertise of all participants” [29].
According to the Merriam-Webster’s online dictionary, the word “consensus” is defined as 1 a.
general agreement, unanimity, b. the judgment arrived at by most of those concerned, or 2.
group solidarity in sentiment and belief. In previous literature on Japanese Delphi surveys, theword “consensus” is used not only as “general agreement” and “judgment arrived at by most of
those concerned” on future direction of technology development but also “group solidarity insentiment and belief” so as to achieve the development of prioritized science and technology
topics as a result of Delphi survey.
Second, as stated above, on what kind of information basis does the evaluation of Delphi
studies in Japan in previous studies reach to such a positive picture? Practitioners’ analyses aremostly based on their own experiences, which are valuable but tend to be subjective or worse
be biased towards being too positive to look good. Works by academics draw on the information
gleaned through interviews with practitioners and users of the results of foresights including6policymakers or industries. Third, previous studies on foresight gave us the impression ofdeterminism, that is, the impression that the result of foresight more or less determines the
allocation of funding, behavior of scientists and engineers, decisions made by government andfirms, direction of science and technology development, and ultimately long-term prosperity of asociety. But even if foresight provides valuable information and opportunities for networking, it is
one of the information sources and one of networking opportunities. It is important to know the
relative size of “behavioral additionality” [30] foresight bring about compared with othermeasures of informing science and technology policy.
Considering those, it is important to know how domain experts perceive the effect of the resultsof foresight activities on the direction of science and technology development in their domain.
Previous literature focusing on Delphi in Japan does not provide an answer to this question fully.
Fig. 1 is a conceptual and simplified model on the impact of foresight on allocation of resources
such as budget, S & T workforce and S & T accomplishment proposed by Eto [28]. The basic
thinking is, as stated above, the foresight in Japan achieves consensus among domain experts,
government policy makers, and industry on promising areas of R & D for a mid- to long-term. As
a result of such consensus, the allocation of resources is concentrated in those promising areas,
which facilitates the accomplishment of S & T goals that the results of foresight set as targets.
So this figure represents the positive evaluation of Japanese foresight in previous studies as
explained above.
Relevant to this study is that “consensus on important topics and accomplishment date”, “S & T
resources” and “accomplishment of important topics” in the figure are consensus among domain
experts, resources allocation towards domain experts, and accomplishment by domain experts.
Therefore, domain experts who participated in foresights are in a good position to judge thedegree to which the practice and results of foresight in Japan gave impact on the resource
allocation, their research environment, and their accomplishment, as compared to other factorsthat may influence those. As stated above, practitioners of foresight is in a position to tend tohave an incentive to tell outside observers that foresight provides valuable input to policymaking
but participated domain experts do not have an incentive to say so to justify their participation.
The model in this figure is not a model that this study intends to show or refute its validity, but astarting point to show to what degree the conceptual model in this figure is valid in describing thesystem for the production of S & T knowledge in Japan to domain expert’s mind.
[Insert Figure 1]

2 Methodology
2.1 Five foresights under study
Perceptions on the impact of foresight of scientists and engineers who participated in major
foresight activities in Japan are measured. Activities include the 6th Delphi survey in 1995/1996,
7

7th Delphi survey in 1999/2000, 8th Delphi survey in 2003/2004, “Social Vision toward 2025”
conducted by NISTEP in 2006/2007, and Science Council of Japan foresight in 2006/2007.
Table 1 compares these foresights. For example, in the 6th Delphi survey, one steeringcommittee with 14 experts was set up, which decided on 14 technological areas(1.material/process, 2.electronics, 3.information technology, 4.life sciences, 5.space,
6.marine/earth sciences, 7.resources/energy, 8.environment, 9.agriculture,
10.production/machinery, 11.architecture/civil engineering, 12.communication technology,
13.transportation, and 14.health and medical sciences). For each technological area, atechnology panel with about 5 to 10 members was set up and it decided on science andtechnological topics in each area, that are expected to be realized within 30 years. A
questionnaire was prepared to determine the degree of importance of each topic, time frame forrealization, Japan’s relative level on the topic compared to Europe, United States and Asia,
necessary government measures for realization, and possible negative impact of topic, and soon. Panel members selected experts to whom the questionnaires could be sent [22]. The 6th
Delphi survey covered 1,072 topics classified into 14 science and technology areas. The firstquestionnaire was sent to 4,868 experts in various science and technology fields, and 4,220
responses were received. The second questionnaire was sent to 4,196 experts, who had
responded to the first questionnaire, and 3,586 responses were received [25; 31]. The basic
format of the 7th and 8th Delphi surveys was the same as in the 6th survey, althoughtechnological areas or questions to be asked were updated and revised [32; 33].
Both Social Vision towards 2025 and Science Council of Japan foresight were prepared as input
to the cabinet level deliberation on “Innovation 25” [34; 35; 36]. Those foresights did not conduct
Delphi surveys and mainly was based on deliberation by panel members, although the former
used the results of the recent Delphi surveys. Innovation 25 was a policy initiative initiated bythen Prime Minister Abe and was intended as a long-term strategy initiative for the creation of
innovation contributing to the growth with an eye on the year 2025. NISTEP and the Science
Council of Japan prepared foresights separately. The Science Council of Japan is a specialorganization under the jurisdiction of the Prime Minister and consists of 210 members, most ofthem academics.
[Insert Table 1]

Fig. 2 depicts the major development of science and technology policy and foresight exercises.
In 1995, Science and Technology Basic Law was enacted in Japan, and every five years,
Science and Technology Basic Plan was prepared based on the law. The First Science andTechnology Basic Plan was decided in July 1996. Reasoning based on sequences of events,
because the second round of 6th Delphi survey was conducted in December 1996 [31], it was
not possible for the 6th Delphi survey to have an impact on the 1st Basic Plan as this wasformulated in July 1996. Likewise, as the second round of the 7th Delphi survey was conducted8

in December 2000 [32], it was difficult for the 7th Delphi survey to have much impact on the 2nd
Basic Plan that was decided on March 2001. However, it was enough time for the 8th Delphi
survey to have impact, because its results were obtained in fiscal year 2004 [33], and the third
Basic Plan was decided in March 2006. It was possible for both kinds of foresights for“Innovation 25” to have an impact on the decision on the content of “Innovation 25”, and wasintended to do so. According to NISTEP, there was a “moderate link between S & T policy andforesight” during the 1990s, and a “link between S & T policy and foresight” during the 2000s[37]. In 2001, the Council for Science and Technology Policy was established within the Cabinet
Office in order to make the decision-making process more top-down and strengthen prioritizationin government R & D funding [21].

[Insert Figure 2]

2.2 Interview
Interviews were conducted with key members of panel committees on Delphi surveys andinnovation 25-related foresights. The selection of interviewees was not random. Interviewees
were chosen by consulting with the person at the Institute for Future Technology who was
responsible for the operation of the 6th, 7th, 8th and Social Vision 2025. The participants in thepanel committees of the surveys who were involved in the deliberation process in terms ofattendance and contributed to the discussion were selected. The requests for interviews weresent basically by e-mails. Interviews were conducted in July and August 2008 and the length of
an interview is about 1 hour.
For each of the five foresights, the author of this article had interviews with two to threemembers (one for Science Council foresight). In addition, the results of interviews are used tocomplement and interpret findings from the survey of this study. In interviews, the following
questions are asked:
What was your role in foresight?

Do you think that the result of foresight is utilized by policymakers or firms?



Do you know any examples of impact of foresight on policy making?

Do you think that the current level of impact of foresight is adequate?

What do you think is necessary to increase the size of impact?

The design of the questionnaire of the survey for domain experts who participated in foresights
was based on the qualitative information collected from interviews, which is explained next.

2.3 Survey
9Panel members, survey respondents, and participants in workshops for the five foresights listed
in Table 1 were asked about their perceptions on the impact of those foresight studies on policy
making. Internet-based survey was conducted. E-mail addresses were collected by searching
the web by participants’ name and organization. I sent e-mails to participants, and they entereddata online. The following is the list of questions:
Q1. Type of organization at the time of foresight (private firm, university, government research
institute, nonprofit organization, other)
Q2. Age at the time of foresight (less than 40, 40 to 49, 50 to 59, more than 60)
Q3. How involved in the foresight (member of steering committee, member of panel committee,
respondent to Delphi survey)
Q4. (For a member of panel committee) Area of panel committee (IT, life sciences, energy, …)
Q5. (For respondents to Delphi survey) Science and technology area(s) (IT, life sciences, energy,…)

2

Q6. Perception on level of impact of the foresight to the following organization (Choose from 1(weak) to 5 (strong)) (central government, regional government, public research institute,private firms, foreign government, individual researchers)
Q7. Perception on level of impact of the foresight with regard to the following type of impact(effective background study for policy making, contribution to making of mid- to long-term
policies, contribution to making of an annual policy, contribution to the decision onprioritization of R & D, contribution to the making of a new R & D program)
Q8. Perception on the level of impact of the foresight with regard to the following policy areas(training of R & D personnel, strengthening relationship among2

For participants in the 6th Delphi survey, the following categories were used: 1.-6. (same as
Q4), 7.Resources/energy, 8.Environment, 9.Agriculture, 10.Production/machinery,
11.City/architecture/civil engineering, 12.Communications, 13.Transportation, and14.Health/medical/social welfare. Those are the names of S & T areas used for Delphi
questionnaires. Those are divided into the 8 categories in the same way as in Q4 except for thefollowing: 7.Resources/energy: Energy, and 8.Environment: Environment.
The following categories were used for the participants in the 7th Delphi: 1.-5. (same as Q4),
6.Maritime/earth, 7.Space, 8.Resources/energy, 9.Environment, 10.Materials/processes,
11.Manufacturing, 12.Distributions, 13.Business Administration, 14.City/architecture/civilengineering, 15.Transportation, and 16.Service. Those are matched as follows: 6.Maritime/earth:
Frontier, 7.Space: Frontier, 8.Resources/energy: Energy, 9.Environment: Environment,
13.Business Administration: Manufacturing, 16.Service: Manufacturing. The rest is the same as
Q4 for the 7th Delphi participants.
For the participants in the 8th Delphi, the Social Vision 2025 and the Science Council foresightstudy, Q5 used the same categories as Q4.
10

industry/universities/government, constructing R & D infrastructure, R & D investment, makebetter environment for private firms, deregulation, introduction of new regulation)
Q9. Perception on overall impact of the foresight
Q10.

Whether the current level of impact of the foresight is adequate

Q11.

(If you think that the current level of impact is inadequate) Reason why it was not possible

to achieve the appropriate level of impact (timing of the foresight, problem in method, notenough efforts at the government to give impact, government not interested, private firms notinterested, the general public not interested, others) (choose one plus open question)
Q12.

Degree of following the situation in general to see if there has been any impacts of theforesight on government policies or other areas (not at all, not very often, neither, moderately
often, and often)
Q13.

How to increase the impact of foresights (open question)

Q14.

Whether interested in receiving the report of this study on impact of foresight in JapanFor Q4, the list of panel committees shown in Table 2 is used for each of the foresights. Eachpanel committee is matched with one of the following 8 science and technology categories foranalysis purpose: 1. ICT, 2. life sciences, 3. nano/material, 4. environment, 5. manufacturing, 6.
infrastructure, 7. frontier technology, and 8. energy.

[Insert Table 2]If participants did not respond after three weeks, a reminder was sent by e-mail. The survey
results are summarized first followed by statistical analysis of the data.

3 Result
3.1 Interview results
Overall, the majority of observations by members of the foresights are that the impact of thosestudies is not very high, and is different from the picture that a consensus on long-term R & D
target is made, results of foresights influence the resource allocation, and foresight guarantee
the nurturing of the S & T strengths. Most think that government has problems in utilizing theresults for policy making. Some think that there is a limitation in Delphi methodology, which does
not produce concrete proposals or measures for policy making, although they provide general
direction. As to the two foresights related to “Innovation 25”, all interviewees think that the results
were used by the report of “Innovation 25.” Other comments are as follows:11

Results of Delphi may be useful for understanding the big picture. But it would be a rare case where
Delphi results were converted into a specific policy initiative. [member of the 6th Delphi]The purpose of conducting Delphi survey is not clear. The survey starts without thinking much about
how to use the results. [member of the 6th Delphi]The quality of the Delphi study is good, but the problem is how to present the results to governmentor the public. [member of the 6th Delphi survey]

Japanese policy does not change by the results of Delphi survey. There is a system to authorize
various reports like Delphi, but no system to utilize them for policy making. [member of the 7th Delphi]

Delphi survey should be conducted by or in close cooperation with organizations that make a strategy.
In the current system the results are not utilized very well. [member of the 7th Delphi]The results of Delphi were not used by government and realized as policy. In Japan, studies like
Delphi is just shelved and not utilized very much. I am very discouraged after participating in theDelphi study. [participants of the 7th Delphi survey]In the Delphi survey the emphasis is on when topics will be realized and not on what the process ofrealization of topics would be like. It is not clear in Delphi how the technology is converted into social
benefits. Without such information, it would be difficult to use the results for policy making. [memberof the 8th Delphi]

Delphi results suggest only the general direction of future science and technology. The general
information does not contribute much to policy choices, which is usually made drawing on very
specific and quantitative information. [member of the 8th Delphi]In the field I participated, there was no impact of Delphi on the 3rd Science and Technology Basic
Plan. [member of the 8th Delphi]

The impact of the 8th Delphi on the 3rd basic plan is not large. There are many studies bygovernment ministries, such as the Ministry of Education, the Ministry of Economy, Trade andIndustry, or the Ministry of Agriculture. Those studies are much more detailed than Delphi andtailored to policy questions more specifically. Delphi is a dictionary-like document, and does not leadto policy directly [member of the 8th Delphi].

In foresight on “Social Visions towards 2025,” we had a very clear goal to have impact on “Innovation
25.” [member of Social Visions towards 2025 foresight]I do not think that results of the Delphi survey, which is based on mail-based survey, is not very
trustworthy. I think that study by experts on a narrower area, for example, by using technology
roadmapping method, is more trustworthy. [member of Social Visions towards 2025 foresight]

The results of foresight by the Science Council of Japan were used for “Innovation 25” [member of
Science Council of Japan foresight].

12

Although those comments are interesting, they are more or less subjective observations
collected from a limited number of interviews I conducted. So in order to examine the averageand distribution of subjective observations among domain experts who participated in theforesights, I conducted a survey and explain the result in the next section.

3.2 Survey results
Table 3 shows the response rate of the survey, which is calculated as the ratio of responses tothe number of e-mail addresses working to which a questionnaire was sent.

[Insert Table 3]

3.2.1 Description of survey results
3.2.1.1. Characteristics of respondents (Q1 and Q2)Table 4 shows that 60%, 61% and 43% of respondents to this survey who participated in the 6th,
7th and 8th Delphi surveys respectively belonged to universities, and 19%, 28%, and 31% of
respondents respectively belonged to private firms at the time of the Delphi survey. Table 5shows that 37%, 42% and 45% of respondents to 6th, 7th, and 8th Delphi surveys respectively
belonged to universities, and 36%, 31% and 27% of respondents belonged to private firms at thetime of the Delphi survey. The proportion of researchers at universities is higher for the foresightby the Science Council of Japan, which is an organization mainly for university professors.
Regarding age distribution at the time of the foresights, less than 20% of the respondents to thissurvey were less than 40 years old and about 10% of respondents were more than 60 years old.
Respondents for the Science Council of Japan foresight were older than those of Delphi surveys.
A comparison of Table 4 and Table 5 shows respondents belonging to universities are larger inthis survey than in Delphi surveys. This is because e-mail addresses of university professors areeasier to locate on the web than those of employees at private firms and government research
institutes. As to age structure, respondents in the 40-49 age bracket is larger compared to 50-59
age bracket, especially for participants in the 6th Delphi survey. This is because part of theparticipants over 50 years old at the time of the survey (1996) are now retired and their e-mailaddresses could not be obtained.

[Insert Table 4 and Table 5]

3.2.1.2. Impact (organization, policy tools, and policy areas) (Q6 –Q8)

13

In Table 6, Table 7 and Table 8, respondents’ perceptions on the size of impact of the foresights
on various aspects of policy making are shown. The score in the figures are calculated using theformula:Score = {(No. of Weak) x 1+( No. of Moderately weak)x 2+( No. of Neither weak nor
strong)x3+( No. of Moderately strong)x4+( No. of Strong)x5}/ (No. of Respondents)
Therefore, a score less than 3 signifies that the extent of impact is weak, and a score greaterthan 3 signifies the extent of impact is strong. Table 6 shows the extent of impact with regard to
the type of organizations affected; the majority of the scores for all of the choices are less than 3
and the extent of impact is perceived to be weak. Among those choices, the score for the
impacts toward central government and public research institutes are relatively higher, but stillless than 3 or slightly above 3. What is interesting is that the score for individual researcher isabout the same as central government and public research institutes. Table 7 shows theperceived extent of impact with regard to policy tools; the score for all categories is less than 3
or slightly above 3, but among those, scores for “cooperation among firms, universities andgovernment” and “R & D funding” are relatively higher. In Table 8, the scores for “contribution todecision on R & D priorities” and “contributions to mid- to long-term policymaking” are higher
than for other categories and are more than 3.
In those tables, the scores for Social Vision toward 2025 foresight are higher for most of thecategories than those for the Delphi surveys, and the scores for the Science Council of Japan
foresight are lower for most categories than those for the Delphi surveys.

[Insert Table 6, Table 7 and Table 8]

3.2.1.3. Overall perception on level of impact and its causes (Q9-Q11)
Overall extent of impact is perceived to be not very strong (Table 9). For example, more than
40% of participants in the 6th Delphi survey think that the level is moderately weak or weak, and
45% think it is “neither strong nor weak.” Eleven percent perceives the level to be moderately
strong or strong. In addition, the level of impact is perceived to be insufficient by almost 80% of
respondents (Table 10). Table 11 shows that respondents think that the low level of impact offoresights in Japan is caused by low level of efforts by the government to make an impact onpolicy and the low level of interest of the general public in the results of the foresights. About 5060% of respondents chose those as the causes. Although the proportion of respondents who
think that the cause is the methodology used for the foresights is less than 5%, up to 30% of
respondents perceive that the results of foresight are not clear.
Fig. 3 compares the distribution of the size of overall impact in five foresights in the survey. The
score (average of impact) gradually increases in Delphi surveys, and the score of “Social Vision14

towards 2025” is higher than Delphi surveys. However, these differences are statistically notsignificant as the 95% confidence intervals of the average are not small enough compared to thedifferences.
[Insert Table 9, Table 10, and Table 11][Insert Figure 3]

3.2.2 Statistical analysis
The effect of various factors on the size of perception of “impact” is analyzed by logistic
regression analysis. Independent variables in the regression, that is, factors that affect thedependent variable (=impact), include the following:Type of foresight (6th, 7th, and 8th Delphi; Social vision toward 2025, and Science
Council of Japan foresight)



Occupation (university, firms, government, and nonprofit)Age (-39, 40-49, 50-59, and 60+)

Manner of involvement in foresight (member of a panel committee or not)

Science and technology areas (ICT, life sciences, nano/material, environment;
manufacturing, social infrastructure, frontier (space or marine sciences), and energy),and

Degree of following if there has been an impact of the foresight on government policies
or other areas (not at all, not very often, neither, moderately often, and often).

The variables are chosen based on Fig.1, since 1) different foresights have different levels of
impact, 2) different domain experts observe different levels of impact, and 3) different levels of
involvement in foresight and different levels of interest in impact of foresight on policy affect theperception. In addition to the analysis on the effects of those factors on 1) the extent ofperceived impact, an analysis is made of the effects of those factors on 2) the perception onwhether the extent of impact is adequate or not. As there are five answer categories to evaluate“impact” in the survey (weak, moderately weak, neither weak nor strong, moderately strong, andstrong), “logistic regression with ordered category (ordered logistic regression)” [38] (pp.911912), [39] (pp.241-244) is used for the first analysis. As the variable on whether the extent of
impact is adequate or not is dichotomous (yes or no), logistic regression analysis is used for thesecond analysis instead of ordered logistic regression.
The characteristics of independent variables used in this analysis are examined (Table 12). All
independent variables for this analysis except for the 6th variable above are dichotomous (0 or1). Table 12 shows the number of each dichotomous variable that takes the value 0 or 1 in each15

of the five foresights and in total (pooled data). For example, the number of the variable“university,” which shows occupation of respondents, equal to 1 (yes) is 310 out of 561
respondents in total, and 67 out of 111 respondents from the 6th Delphi survey. As to the kinds
of foresights, the majority of respondents are participants in the three Delphi surveys, andrespondents of participants in the “Social Vision toward 2025” and Science Council of Japan
foresight are relatively small. As to science and technology areas, all eight areas are
represented in the respondents, although there are no respondents in some of the areas in“Social Vision toward 2025” foresight as the result of small number of committees (six) in thisforesight (see Table 2). The eight science and technology areas are areas used in the second
and third Science and Technology Basic Plans. Information and communication technology(ICT), life sciences, nano/material, and environment are designated as “primary prioritized areas”
and manufacturing, infrastructure, frontier technology, and energy are designated as “secondaryprioritized areas” [40]. As there are more than eight science and technology areas used in
Delphi surveys, areas in Delphi surveys are matched with those eight science and technology
areas.
Table 13 shows the distributions of two above-mentioned dependent variables, which will be
discussed in the regression analysis.

[Insert Table 12 and Table 13]

Table 14 shows the result of statistical analysis on the effect of various factors (foresight,organization, age, member status, science and technology area, and the degree of following thesituation) on a) the perception on impact, and b) perception on whether the level of impact isadequate. Pooled data including the data on respondents who participated in the 6th Delphi
survey, 7th Delphi survey, 8th Delphi survey, Social vision towards 2025, and Science Council ofJapan foresight was used for the analysis.
3.2.2.1 Impact
Ordered logistic regression is used for this dependent variable. Ordered logistic regression
estimates a score, S, as a linear function of independent variables. For example, when a
respondent participated in the 8th Delphi (base group in foresight category) and he was auniversity professor (base group in occupation category) in his forties and not a panel member,and follow if there has been any impacts neither often nor not at all, and his academic
background is nano/material area, the score is calculated as:
S = 0+0-0.52+1.01+1.88 x 0.5=2.46
Because this is between cut point 2 and cut point 3, it is most likely that the respondent’s
perceived impact is “neither weak nor strong.” When the score moves higher, it becomes more16

likely that respondent’s selection moves to a stronger level of impact. Significant variables are
firms (+ (meaning sign is positive compared with a base group, or university in this category)),
non-profit (+), age less than 40 (-), age 40-49 (-), life science (+), nano/material (+), and degreeof following (+), which shows the following:Survey respondents at private firms or at nonprofit organizations perceived a larger impactof foresight they participated than respondents at universities (base group), controlling forother factors.

Survey respondents in their twenties, thirties, and forties perceived a lower level of impact
than respondents in their fifties (base group). In other words, younger respondents saw less
impact, controlling for other factors.

Survey respondents in life science or nano/material perceived a larger level of impact thanrespondents in other science and technology areas, controlling for other factors. It is
understandable that participants in primary prioritized science and technology areas in the
Science and Technology Basic Plan perceived more impact than in other areas. In thesecond basic Plan, science and technology fields are prioritized, and in the third Basic Plan,
topics within science and technology fields are prioritized in more detail.



Survey respondents who follow more if there have been impacts perceived more level ofimpact.

However, the effects of those factors are not very strong because the sizes of coefficients are
smaller than the size of score between the cut points. In other words, it is difficult to accumulatea score to go beyond the cut points 3 and 4.
3.2.2.2. Is impact adequate?
As the answer is dichotomous (yes or no) in the question, logistic regression is used. If thecoefficient is positive, it means that the increase of the independent variable increases the odds
of choosing the answer “yes” over “no.” Full model does not fit the data. Instead, only agerelated variables and member variable are used. Likelihood chi squared is 9.65, and is
statistically significant with 95% confidence. Significant variables are age 40-49 (+), and member(-), which shows the following:Survey respondents who are a member of the foresights are more likely to perceive that thecurrent level of impact is inadequate than non-member, controlling for other factors. It is
understandable that members of foresights who are more committed to foresight expect
more impact from it and set a higher standard for adequateness of impact.

[Insert Table 14]17

3.2.3 Perception and reality
There may be critical argument that the domain experts’ responses to the survey are based juston their perception, which may not fully reflect upon the reality on the foresights’ past impact onpolicy making. After all, domain experts are experts on their S & T domains, and not experts onpolicymaking, although they are in an R & D environment directly or indirectly influenced bypolicy decisions on resource allocation. Perception is defined as the “process by which people
select, organize, interpret, retrieve, and respond to information from the world around them” [41].
Or, perception is defined as the “process of organizing and interpreting sensations intomeaningful experiences” and “the result of psychological processes in which meaning, context,
judgment, past experiences, and memories are invoked” [42]. So expert’s past experiences ormemories affect more or less which information they select or respond, or how to interpret them,and if such effects are too strong to cloud the real picture, his judgment or perception of thereality is called biased or prejudiced. Perception is a “dynamic conflict between the attempts ofan outer world to impose an actuality on us and our efforts to transform this actuality into a selfcentered perspective [43].”
In order to look at the degree of “perceptionness” of the responses to this survey, that is, thedegree of effects of an “outer world” on the process of cognition, two factors, that is, the degree
to which the respondent follow if there has been any impacts and whether the respondent is apanel member or not, are examined. It is expected that “attempts of an outer world to impose an
actuality” is stronger if a respondent follows the situation around them more often, and is a panel
member. So there should be a trend in the responses related to those two factors.
The results of statistical analysis shown in Table 14 show the following. First, the effect of “panel
member or not (variable name: “member”) on their perception on overall impact is notstatistically significant, but the effect of “member” on the degree that they feel the effect isadequate is statistically significant and negative. That is, being a panel member does not affecttheir perception on the size of overall impact of foresight on policymaking but respondents who
was a panel member tend to perceive that such size of impact is not adequate enough. Second,the effect of how often they follow if there have been impacts (variable name: “follow”) on theirperception on the size of impact is statistically significant and positive. That is, more often they
follow, larger their perception on the size becomes.
Next, the data is examined in terms of the two variables, “member” and “follow” in more detail.
Table 15 shows the mean and 95% confidence interval of the mean for each category of
responses in terms of question on 1) panel member or not, and 2) how often you follow the news
related to impact. In general, when comparing answer categories to the question on “follow,” themean for the panel members is higher than mean for the non-members, and mean forrespondents who follow the situation more often tends to be higher. For example, whencomparing the mean for the respondents who are panel members and who follow the situation“very often” and the mean for respondents who are not panel members and who do “not at all”
18

follow the situation, the former is higher and the difference is statistically significant. In addition,
the mean for the respondents who are not panel members and “do not all” follow the situation islower than the mean for the respondents who are not panel members and who do follow thesituation “not very often” “neither” or “often” and the differences are statistically significant.
However, for other pairs, the difference is not statistically significant. In other words, 95%
confidence intervals of the mean tend to be overlapping because Ns for some categories are notvery high.

[Insert Table 15]

From those data, the influence of “attempts of an outer world to impose an actuality on us” isseen to a certain degree and a certain degree of “perceptionness” is seen in the responses. In
other words, more information from outside on the impact of foresight on policymaking, that is,
being a panel member or following more often, affects their perception on the size so as to makeit larger. So the size of impact of foresight on policymaking in the responses to this survey isbiased downwards to a degree when comparing it with “reality.”
What is the size of the degree of bias downwards? The mean response from the respondents
who are panel members and who follow the situation “very often” is 3.14, in other words, near tothe category “the size of impact is neither strong nor weak.” On the other hand, the mean
response from the respondents who are not panel members and who do not at all follow thesituation is 2.07, in other words, near to the category “the size is moderately weak.” So even if
there is a certain degree of bias, and assuming that respondents in the former category is morebias-free and the respondents in the latter category is more biased, getting rid of all the bias is
not enough for moving the perceived size of impact higher up to the level of “strong” or “verystrong.”Having said that, it is important to acknowledge that there is a case where perceived reality ismore important than the reality itself. For example, when the role of disorder is examined forknowing the process of the social differentiation of urban areas and neighbourhood change, youfind that perceived disorder plays a vital role, since “it is not a disorder but people’s assessmentof the seriousness of disorders that matters [44].” In the case of technology foresight, how isperception on the impact on policy-making more important than the reality or as important as thereality? There would be two possibilities. First, participation rate of scientists and engineers inforesight, or response rate to Delphi surveys and attendance of panel members would beenhanced if they perceive that their activities are important for policy-making. Second, quality of
information provided by scientists and engineers who participated in foresight, or efforts and time
allocated for contributing to the product of foresight would be enhanced if they perceive theiractivities are important for policy making. Although the corroboration of those points, or19hypotheses, is beyond the scope of this study, and the relationship of various factors that could
affect the participation rate or quality of participation is clearly very complex, after all, the “mostimportant resource input (for foresight) is voluntary contribution of participants” [30], andmotivation other than pecuniary one for them is important for increasing frequency and quality of
their participation.

4 Conclusion and policy implications
4.1 Key findings
There are two main findings of this study. First, among the domain experts who participated inforesight in Japan, the level of perception on its impact on policy making is found to be not very
high, based on the results of interviews and surveys conducted on the five foresights in Japan.
In addition, the majority think that the current level of impact is inadequate. Although this finding
seems to be different from previous literature about foresights in Japan, which tends to interpretthe role of foresight more or less as critical information basis for S & T policy and technologicaldevelopment in Japan as shown in Figure 1, it is necessary to be cautious in interpreting theresults as representing the “reality” on the impact of foresight on policy making in Japan because
this is based on experts’ perceptions or views on it. However, at least the results reflect thegeneral situation on the role of foresight for policy making in Japan and the perception, which
may not be fully mirror the reality, has the significance for itself for the conduct of foresight interms of motivation of domain experts for participating in foresights.
Second, it is found that there are different patterns of perception on impact on policy making(including adequate or not) by ages, organizations, member status, and science and technology
areas. Experts in prioritized science and technology areas naturally perceive more impact. Panelmembers expect more impact and are less satisfied. Experts working for private firms think thatthe impact is larger than university professors, but may have different (or lower) expectations
from the result of foresight.
Generalizability of the finding across foresight in general is not proved in this study that analyzed
only foresights conducted in Japan between 1996 and 2007. So the following is the speculative
conjecture. Two general questions can be asked from the findings of this study.
1) Does the level of domain experts’ perception on impact of foresight on policy tends to belower than methods experts or users?
2) If so, does that come from the fact that domain experts have different perspectives on theimpact of foresight on policy from methods experts or users?
Considering the finding from this study that domain experts’ perceptions are not very high inspite of the fact that previous research shows the high impact of Japanese foresight on policy, it20

is probable that there is more or less gap between the perception between domain experts and,methods experts and users. As to the second question, one study that examined the impact ofpolicy evaluation research on policy in the field of public health pointed out that “We found that
evaluation research is used by decisionmakers but not in the clear-cut and organization-shaking
ways that social scientists sometimes believe research should be used” and “The problem ...
may well lie more in many social scientists’ overly grand expectations about their own
importance to policy decisions than in the intransigence of federal bureaucrats.” Then theauthors concluded that “what is typically characterized as underutilization or nonutilization ofevaluation research can be attributed in substantial degree to a definition of utilization that is too
narrow and fails to take into consideration the nature of actual decision-making processes inmost programs”[45].
This may be a probable explanation that is also valid to the domain experts’ perception onforesight. However, even if this explanation is valid for explaining the gap in perceptions, that
does not lead to the conclusion that it is ok to leave domain experts to continue to have thatnegative perception after participating in foresight. Finally, going back to the Japanese foresightcase, even if domain experts in Japan have “overly grand expectations,” it is still unexpectedthat they have on average the perception that the impact of foresight in Japan on policy is notvery high, considering the very positive picture on the various kinds of good effects of Japanese
foresight depicted in the previous literature.

4.2 Limitation of the study
There are three sources of limitations of this study. First, response rates are not very high, which
may lead to inaccuracy or bias of estimates. In addition, respondents may not be representativein this e-mail-based survey, especially for older foresight activities, although the similarity of thedata on respondents and original participants of foresights was checked. Second, there is thelimitation of memory of participants. Do participants remember what they did or heard more than
10 years ago? Although it is important to stimulate the memory of participants on foresights in asurvey, it certainly decreases the response rate if too much is demanded, for example, by asking
participants to read the foresight they contributed to produce. Third, the fact that this study
analyzes “perception” of domain experts is the limitation of this study if one would like tomeasure the impact of foresight on policy making from the viewpoint closer to the policy making
process and not through the more or less subjective eyes of domain experts. Perception of
domain experts may not be the reality in all instances.

4.3 Policy implication
One implication of the study is that the impact is perceived differently among domain experts
who participated in foresight. If it is necessary to discuss the impact of foresight during theevaluation stage of foresight activities, it is better to collect observations and views of domain
experts in various organizations, science and technology areas, member status, or age groups.
21

Another policy implication is that more feed-back information to domain experts on how theresults will be used and were used would be necessary, to maintain their motivation inparticipation. Scientists and engineers are interested in knowing how they could make adifference in policy making by providing their expertise. Provision for more feed-back information
would improve the involvement of participants and increase the quality of foresight.

Acknowledgement: This study was partly funded by the Watanabe Memorial Foundation for the
Advancement of Technology. The author appreciates the helpful comments of two reviewers,
and financial support from the European Commission for presenting a part of this study at aconference at the Institute for Prospective Technological Studies.

22References
1. Technology Futures Analysis Methods Working Group. Technology futures analysis:
Toward integration of the field and new methods. Technology Forecasting and Social Change.
71, pp.287-303. 2004.
2. Martin, B.R. Foresight in science and technology. Technology Analysis and Strategic
Management 7, 139–168. 1995.
3. Woudenberg, Fred. An evaluation of Delphi. Technological Forecasting and Social Change
40, pp. 131–150. 1991.
4. Rowe, Gene; Wright, George; Bolger, and Fergus. The Delphi technique as a forecasting
tool: issues and analysis. International Journal of Forecasting. Vol.15, Issue 4, pp. 353-375.
1999.
5. Cuhls, Kerstin. Futur – foresight for priority-setting in Germany. International Journal ofForesight and Innovation Policy. Vol. 1, No.3/4. Pp.183–194. 2004.
6. Cuhls, Kerstin, and Luke Georghiou. Evaluating a participative foresight process: ‘Futur the German research dialogue.’ Research Evaluation. Vol. 13, No.3, 1 December 2004.
Pp.143-153. 2004.
7. Martin, Ben R. and Ron Johnston. Technology Foresight for Wiring Up the National
Innovation System – Experiences in Britain, Australia, and New Zealand. Technological
Forecasting and Social Change, Volume 60, Number 1, January 1999, pp. 37-54. 1999.
8. Rask, Mikko. Foresight – balancing between increasing variety and productive convergence.
Technological Forecasting and Social Change. 75 (2008) 1157-1175. 2008.
9. Blind, Knut, Kerstin Cuhls and Harolf Grupp. Current Foresight Activities in Central Europe.
Technological Forecasting and Social Change. 60, Pp.15-35. 1999.
10. Shin, Taeyoung. Using Delphi for a Long-Range Technology Forecasting, and Assessing
Directions of Future R & D Activities: The Korean Exercise. Technology Forecasting and Social
Change. 58. Pp.125-154. 1998.
11. Grupp, Hariolf, and Harold A. Linstone. National Technology Foresight Activities Aroundthe Globe: Resurrection and New Paradigms. Technological Foresight and Social Change. 60.
Pp.85-94. 1999.
12. Héraud, Jean-Alain, and Kerstin Cuhls. Current Foresight Activities in France, Spain, andItaly. Technological Forecasting and Social Change. 60. Pp.55-70. 1999.

23

13. Anderson, Joe. Foresight analysis in research funding agencies: A UK experiment. Higher
Education. 28: Pp.39-54. 1994.
14. Irvine, J. and B.R. Martin. Foresight in Science: Picking the Winners. London/Dover :Frances Pinter, 1984.
15. Munier, Francis and Patrick Ronde. The role of knowledge codification in the emergenceof consensus under uncertainty: empirical analysis and policy implications, Research Policy, 330(2001), 1537-1551. 2001.
16. Kondo, Satoru. R & D Senryaku Ritsuan No Tameno "Gijutsu Yosoku" Katsuyo Guidebook(Guidebook on technological forecasting for making R & D strategy) (in Japanese). Tokyo :Urban Produce, 1993.
17. Keenan, Michael and Ian Miles. Scoping and Planning Foresight in The Handbook ofTechnology Foresight edited by Luke Georghiou et al, pp.342-375. Cheltenham : Edward Elgar
Publishing, 2008.
18. Miles, Ian et al. New Frontiers: Emerging Foresight. In The Handbook of Technology
Foresight: Concepts and Practice. pp.400-417. Cheltenham : Edward Elgar Publishing, 2008.
19. —. The Many Faces of Foresight. in The Handbook of Technology Foresight: Concepts andPractice. Edited by Georghiou, Luke et al. Northampton: Edward Elgar Publishing. 2008.
20. Hanney, Steve, Mary Henkel and Dagmar von Walden Laing. Making and implementing
foresight policy to engage the academic community: health and life scientists' involvement in,
and response to, development of the UK's technology foresight programme. Research Policy 30(2001) 1203–1219. 2001.
21. Kuwahara, Terutaka, Kerstin Cuhls and Luke Georghiou. “Foresight in Japan” in The
Handbook of Technology Foresight: Concepts and Practice, edited by Luke Georghio et al.
Cheltenham : Edward Elgar Publishing, 2008.
22. Kuwahara, Terutaka. Technology Forecasting Activities in Japan. Technological
Forecasting and Social Change, 60, 5-14. 1999.
23. Kondo, Satoru. Method and Application of Technological Forecasting: Centering around theDelphi Method [in Japanese], Japan Society for Science Policy and Research Management,
Vol.7 Issue 1, 1992, pp.20-25. 1992.
24. National Institute of Science and Technology Policy. Outlook for Japanese and German
Future Technology - Comparing Japanese and German Technology Forecast Surveys-. NISTEP
report material No.33. March 1994. 1994.
25. —. The Results of the User Survey on "The 6th Technology Forecast Survey." NISTEP reportmaterial No.69. March 2000. 2000 .
24

26. Vos, Cornelis M. and Cor L. Balfoort. Strategic conferencing: A new approach in science
policy, Research Policy, 18, pp. 51-57. 1989.
27. Cuhls, Kerstin. Technikvorausschau in Japan. Physica-Springer, Heidelberg. 1998.
28. Eto, Hajime. Obstacles to the acceptance of technology foresight for decision makers.
International Journal of Foresight and Innovation Policy. Issue: Vol.1, No. 3-4. Pp. 232 – 242.
2004.
29. Gordon, Theodore Jay. The Delphi Method, Futures Research Methodology, AC/UNU
Millennium Project. 1994.
30. Georghiou, Luke and Michael Keenan. Evaluation and Impact of Foresight. In The
Handbook of Technology Foresight: Concepts and Practice. Edited by Georghiou, Luke et al.
Edward Elgar Publishing. 2008. 2008.
31. National Institute of Science and Technology Policy. The Sixth Technology Forecast
Survey- Future Technology in Japan Toward The Year 2025 -. NISTEP Report No. 52. June
1997. URL: www.nistep.go.jp/achiev/ftx/eng/rep052e/idx052e.html. 1997 .
32. —. The Seventh Technology Foresight: Future Technology in Japan toward the Year 2030.
NISTEP Report No.71. July 2001. URL: www.nistep.go.jp/achiev/ftx/eng/rep071e/idx071e.html.
2001.
33. —. The Eighth Technology Foresight: Future Technology in Japan toward the Year 2035.
NISTEP Report No.97. May 2005. URL: www.nistep.go.jp/achiev/ftx/jpn/rep097j/idx097j.html (inJapanese). 2005.
34. —. Social vision towards 2025: scenario discussion based on S & T foresight. NISTEP Report
No.101. May 2007. URL: www.nistep.go.jp/achiev/ftx/jpn/rep101j/idx101j.html (in Japanese).
2007 .
35. Science Council of Japan. The Future Society Envisioned by the Science Community.
Committee for the Investigation of Innovation Promotion. January 25, 2007. URL:
www.scj.go.jp/ja/info/kohyo/pdf/kohyo-20-t32e.pdf. 2007.
36. Government of Japan. Long-term Strategic Guidelines “Innovation 25” (Unofficial
Translation). June 1, 2007. URL: www.kantei.go.jp/foreign/innovation/innovation_final.pdf. 2007.
37. Kuwahara, Terutaka. A New Multi-Methodology Foresight for Prioritization in the Next
National S&T Basic Plan; presented at KISTEP-WREN Workshop on “National Models for
Public R&D Evaluation: In Search of Best Practices and Collaborative Opportunities” May 31,
2005,. 2005.
38. Green, William H. Econometric Analysis. Third Edition. New Jersey : Prentice Hall, 1997.
39. Hamilton, Lawrence. Statistics with STATA 5. Duxbury Publishing. 1998.
25

40. Council for Science and Technology Policy. Outline for the 3rd Science &Technology
Basic Plan (FY2006-2010), Dec. 27, 2005. URL: www8.cao.go.jp/cstp/english/basic/3rdBasicPlan_outline.pdf. 2005.
41. Schermerhorn, John R., Jr.; James Hunt and Richard N. Osborn. Organizational
behavior, Eighth Edition. New York : John Wiley & Sons, 2003.
42. Schiffmann, H. R. Sensation and Perception: An Integrated Approach, 3rd edition. New
York : Wiley, 1990.
43. Rummel, R.J. Understanding Conflict and War Volume 1: The Dynamic Psychological Field.
Beverly Hills, CA : Sage, 1975.
44. Wikström, Per-Olof H. . “Questions of Perception and Reality.” British Journal of
Sociology. 2009 Vol.60, Issue 1. 2009.
45. Patton, Michael Q. et al. "In Search of Impact: An Analysis of the Utilization of Federal
Health Evaluation Research" in Using Social Research in Public Policy Making. pp.141-163.
Lexington, Massachusetts : Lexington Books, 1977.

26

Consensus on important topics and their accomplishment dates

S & T media

Project priority

Future prospect among S & T workforce

Hunting of promising jobs

Project selection

Job creation

BudgetS & T resources

Accomplishment of important topics
Source: Figure 2 (Effects of Delphi consensus on accomplishment) in Eto, Hajime (2004).
“Obstacles to the acceptance of technology foresight for decision makers.” International Journalof Foresight and Innovation Policy. Issue: Vol.1, No. 3-4. Pp. 232 – 242 is revised.
Figure 1: Conceptual model on impact of technology foresight in Japan on science andtechnology accomplishment

27

Table 1: List of foresight analyzed in this study
Surveyperiod
6th Delphi survey

1995-1996

No. oftechnical
areas
14

No. oftopics

Forecasted
period1072

30 years to2025

7th Delphi survey

1999-2000

16

1065

30 years to2030

8th Delphi survey

2003-2004

13858

30 years to2035

Social vision
towards 2025

2006-20076-

20 years to2025

Science Council ofJapan foresight

2006-2007-

20 years to2025-

Membersof panelgroups129 (1steering
committee,
13 panels)
173 (1steering
committee,
17 panels)
190 (1steering
committee,
13 panels)
93 (1steering
committee,
6 panels)
20 (1committee)

Note: Numbers of responses to the 2nd round of the Delphi survey are shown.

28

No. ofresponses
3586

3106

2239

-

-

Major developmentin Science and Technology Policy

Foresight exercises[Decided in July 1996]
The 1st S & T Basic Plan (1996-2000)

1995-96 The 6th Technology Foresight

1999-2000 The 7th Technology Foresight[Decided in March 2001]
The 2nd S & T Basic Plan (2001-05)

2003-04 The 8th Technology Foresight[Decided in March 2006]
The 3rd S & T Basic Plan (2006-2010)2006-07 Social vision towards 2025
2006-07 Science Council of Japan foresight[Decided in July 2007]
Innovation 25

Figure 2: Foresight and major development of science and technology policy in Japan

29

Table 2: Panel committees set up in each foresight
Panel committees
6thDelphisurvey7thDelphisurvey8thDelphisurveySocialVision2025

ScienceCouncil

8 categories for analysis

1. Materials/Process
2. Electronics
3. Information technology
4. Life sciences
5. Space technologies
6. Marine/Earth
7. Resources/Energy/Environment
8. Agriculture
9. Production/Machine
10. City/Architecture/Civil engineering
11. Communications
12. Transportation
13. Health/Medical/Social welfare
1. Information and communication
2. Electronics
3. Life sciences
4. Health/Medical
5. Agriculture/Food
6. Marine/Earth/Space
7. Resources/Energy/Environment
8. Materials/Process
9. Manufacturing
10. Distributions
11. Business administration
12. City/Architecture/Civil engineering
13. Transportation
14. Service
15. New society/Economic system
16. Declining birthrate/Aging society
17. Safety
1. Information and communication
2. Electronics
3. Life sciences
4. Health/Medical/Social welfare
5. Agriculture/Food
6. Frontier
7. Energy/Resources
8. Environment
9. Nanotechnology/Materials
10. Manufacturing
11. Industry infrastructure
12. Social infrastructure
13. Social technology
1. Health during entire life
2. Information environment as infrastructure fordaily-living
3. Support for people by advancement of brain
sciences
4. Safe and sustainable city
5. Satisfying life: selection of jobs, child rearing,
diversification of senior life
6. Resolution of global environment problems byliving together with the world
1. Information and communication
2. Electronics

30

→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→
→

→
→

3. Nano/Materials
1. ICT
1. ICT
2. Life sciences
7. Frontier technologies
7. Frontier technologies
8. Energy, 4. environment
2. Life sciences
5. Manufacturing
6. Infrastructure
1. ICT
6. Infrastructure
2. Life sciences
1. ICT
1. ICT
2. Life sciences
2. Life sciences
2. Life sciences
7. Frontier technologies
8. Energy, 4. environment
3. Nano/Materials
5. Manufacturing
5. Manufacturing
5. Manufacturing
6. Infrastructure
6. Infrastructure- (no response)- (no response)- (1 response)- (no response)
1. ICT
1. ICT
2. Life sciences
2. Life sciences
2. Life sciences
7. Frontier technologies
8. Energy
4. Environment
3. Nano/Materials
5. Manufacturing
5. Manufacturing
6. Infrastructure
6. Infrastructure
2. Life sciences
1. ICT→

2. Life sciences, 1. ICT→→

6. Infrastructure- (2 responses)→

4. Environment→→

1. ICT
1. ICTforesight

→
→
→
→
→
→
→
→
→
→
→
→
→
→

3. Life sciences
4. Health/Medical/Social welfare
5. Agriculture/Food
6. Frontier
7. Energy/Resources
8. Environment
9. Nanotechnology/Materials
10. Manufacturing
11. Industry infrastructure
12. Social infrastructure
13. Social technology
14. Social science
15. Humanities
16. Others

2. Life sciences
2. Life sciences
2. Life sciences
7. Frontier technologies
8. Energy
4. Environment
3. Nano/Materials
5. Manufacturing
5. Manufacturing
6. Infrastructure
6. Infrastructure- (3 responses)- (2 responses)- (no response)

Note: In the 7th Delphi survey, there were no responses in “14. Service,” “15. New
society/economic system” and “17. Safety.” The 16th category (Declining birth rate/Agingsociety) was not matched with the 8 S & T categories, since this is a category for examining theneeds for science and technology in general. The number of responses for each of those panels
was shown in the parenthesis. For the same reason, the 5th category in the Social Vision 2025,
and the 14th, 15th and 16th categories in Science Council foresight were not matched.

31

Table 3: Number of responses and response rate of the survey
No. ofparticipants(Delphi,etc.)

No. of emails sent

6th Delphi

3586

895

7th Delphi

3106

8th Delphi

No. ofResponses

Response
rate (%)

695

112

16

2723

1562

209

13

2239

1022

693

197

28

Social vision
towards 2025

228

228

225

43

19

Science Council ofJapan foresight

294

119

101

16

16

32

No. of emails
workingTable 4: Characteristics of respondents to this survey (organization at the time ofparticipation in foresight)

Organization

Age

University
Private firm
Government
Not for profit
Others
-39
40-49
50-59
60+

6th Delphi(n=111) (%)

7th Delphi(n=203) (%)

60
19
13
7
1
15
49
25
10

61
28
4
6
1
11
40
37
11

8th Delphi(n=189) (%)

Social vision
2025 (n=42)
(%)
43
52
31
19
18
10
4
10
4
10
16
22
34
24
40
37
11
17

Sciencecouncil 2025
(n=16) (%)
88
6
6
0
0
0
25
44
31

Note: For age, n=202 for the 7th Delphi, n=190 for the 8th Delphi, and n=41 for social vision
2025.
Source: Survey conducted by the author

33

Table 5: Characteristics of respondents to the Delphi survey (organization at the time ofparticipation in foresight)
Organization

Age

University
Private firm
Government
Not for profit
-39
40-49
50-59
60+

6th Delphi (%) 7th Delphi (%)
37
42
36
31
15
14
10
10
9
10
36
31
41
44
14
16

8th Delphi (%)
45
27
19
4
11
33
40
14

Source: National Institute of Science and Technology Policy [31; 32; 33]

34

Table 6: Foresight’s Impact on organizations

Central government
Local government
Public research institutePrivate firms
Foreign government
Individual researchers

6th
Delphi(n=111)

7th
Delphi(n=200)8th
Delphi(n=188)Socialvision 2025
(n=41)

3.0
2.0
2.9
2.7
2.2
2.6

3.1
2.0
3.0
2.7
1.9
2.7

3.1
2.1
3.2
2.5
1.9
2.7

3.5
2.2
3.2
2.7
2.0
2.8

Source: Survey conducted by the author

35

ScienceCouncil2025(n=16)
2.2
1.8
2.7
2.3
2.0
2.7

Table 7: Impact on policy tools

Education/training of researchpersonnel
Cooperation among firms,
universities, and government
R & D infrastructure
R & D funding
Environment for private firms
Deregulation
Introduction of new regulation

6th
Delphi(n=111)

7th
Delphi(n=200)8th
Delphi(n=188)
2.7

Socialvision2025(n=41)
2.9

ScienceCouncil2025(n=16)
2.6

2.6

2.7

2.9

2.9

3.0

3.1

2.9

3.0
2.9
2.5
2.6
2.7

2.9
3.0
2.5
2.4
2.5

2.9
3.0
2.5
2.7
2.8

3.1
3.0
2.2
2.6
2.7

2.3
2.3
2.4
2.3
2.5

Source: Survey conducted by the author

36

Table 8: Foresight’s impact (types of influences)

Effective background researchfor policy making
Contribution to mid-to-longterm policymaking
Contribution to annual
policymaking
Contribution to decision on
research priorities
Contribution to new research
program

6th
Delphi(n=111)

7th
Delphi(n=198)3.0

3.0

Socialvision2025(n=41)
3.0
3.4

2.9

3.1

3.2

3.1

2.7

2.5

2.6

2.6

2.9

2.1

3.2

3.2

3.3

3.3

2.7

3.0

3.1

3.0

3.3

2.9

Source: Survey conducted by the author

37

8thDelphi(n=188)

ScienceCouncil2025(n=16)
2.7

Table 9: Overall level of impact

StrongModerately strong
Neither weak nor strong
Moderately weakWeak

6th Delphi(n=109)(%)

7th Delphi(n=200)(%)

8th Delphi(n=188)
(%)

1
10
45
21
22

0
15
39
36
10

Source: Survey conducted by the author

38

SocialSciencevision 2025 Council(n=41) (%) 2025
(n=15) (%)
1
0
7
18
24
20
44
39
13
26
27
40
11
10
20

Table 10: Is overall level of impact adequate?
6th Delphi(n=104)(%)
Adequate
Not adequate

19
81

7th Delphi(n=200)(%)

8th Delphi(n=185)
(%)

19
81

23
77

Source: Survey conducted by the author

39

Socialvision 2025
(n=41) (%)
17
83

ScienceCouncil2025(n=16) (%)
7
93

Table 11: Reasons for low level impact
6th Delphi

7th Delphi

8th Delphi(n=98) (%)(n=170)(%)(n=156)(%)

Timing of study
Methodology of study
Unclear results of study
Not enough effort at government
to increase impact
Government not interested inresult of study
Firms not interested in result ofstudy
The public not interested in resultof study
Other reason

Socialvision 2025
(n=33) (%)

ScienceCouncil2025(n=14) (%)

0
7
23
51

2
4
30
53

1
4
30
49

3
9
24
48

0
21
14
36

31

33

32

33

57

28

41

31

33

21

57

59

61

61

29

7

8

10

9

21

Source: Survey conducted by the author

40

3.5
3.07

3.0
2.65
2.47
2.28

2.5

2.83
2.70
2.57

2.69
2.57
2.44

2.0

3.16

2.78
2.49

2.53

1.90

1.5
1.0
6th Delphi

7th Delphi

8th delphi

Social Vision Science Council
2025

Note: Marker shows the mean and line shows the 95% confidence interval of the mean, which is
calculated as mean plus minus 1.96 x standard error of the mean.
Source: Survey conducted by the author
Figure 3: Comparison of the mean of perceived size of impact in 5 foresights (average and95% confidence interval)

41

Table 12: Distribution of independent variables (dichotomous - 1: yes, 0: No)

Foresight
Variable

Foresight-related
variables

Occupation(organiza
tion)
Age

MemberS&T
areas

Followthesituation

Delphi 6
Delphi 7
Delphi 8
Social vision
Science Council
University
Firms
Gov
Not for profit
-39
40-49
50-59
60Member
ICTLife sciences
Nano/material
Environment
Manufacturing
Infrastructure
Frontier tech
EnergyNot at all
Not very often
NeitherModerately often
Often

Total
(pooleddata)

6th Delphi

7th Delphi

8th Delphi

Socialvision 2025

ScienceCouncil of
Japan

0

1

0

1

0

1

0

1

0

1

0

1

463
366
378
531
558
251
418
500
530
479
347
359
492
498
417
399
491
437
499
468
489
470
482
339
401
455
539

111
208
196
43
16
310
143
61
31
80
212
200
67
61
143
161
69
123
61
92
71
90
72
215
153
99
15

0
111
111
111
111
44
90
97
103
94
57
82
100
102
78
85
93
87
106
97
86
93
89
76
76
94
109

111
0
0
0
0
67
21
14
8
17
54
29
11
9
33
26
18
24
5
14
25
18
22
35
35
17
2

208
0
208
208
208
78
147
195
191
178
121
127
180
179
151
156
177
156
169
167
173
166
177
121
141
164
197

0
208
0
0
0
125
56
8
12
24
81
75
22
23
50
45
24
45
32
34
28
35
23
79
59
36
3

196
196
0
196
196
107
132
155
182
159
126
115
167
184
154
125
165
147
168
156
174
156
164
112
142
147
183

0
0
196
0
0
82
57
34
7
30
63
74
22
5
35
64
24
42
21
33
15
33
23
75
45
40
4

43
43
43
0
43
20
34
38
38
32
31
26
34
18
24
23
43
35
43
35
43
43
39
16
32
41
36

0
0
0
43
0
22
8
4
4
9
10
15
7
23
19
20
0
8
0
8
0
0
2
25
9
0
5

16
16
16
16
0
2
15
15
16
16
12
9
11
15
10
10
13
12
13
13
13
12
13
14
10
9
14

0
0
0
0
16
14
1
1
0
0
4
7
5
1
6
6
3
4
3
3
3
4
2
1
5
6
1

Note: The table shows the number of each dichotomous variable takes the value 0 (no) or 1 (yes) in eachof the five foresights and in total (pooled data).

42

Table 13: Distribution of dependent variables
Foresight
Variable
ImpactImpactadequate

Weak
Moderately weak
NeitherModerately strong
Strong
Yes
No

Total
(pooleddata)
74 (13.4%)
162 (29.3%)
227 (41.1%)
87 (15.7%)
3 ( 0.5%)
111 (20.4%)433 (79.6%)

6thDelphi

7th
Delphi
24
23
50
11
1
20
84

22
72
77
29
0
40
160

8th
Delphi
21
50
82
34
1
43
142

Socialvision
2025
4
11
16
10
0
7
34

ScienceCouncil
Japan
3
6
2
3
1
1
13

Note: The table shows the number of each dependent variable takes each of the answer categories
in total (pooled data) and in each of the five foresights

43

Table 14: Effect of respondent’s characteristics on perception on impact (pooled data)

Predictors

Impact6th Delphi-0.321

7th Delphi-0.266Social Vision towards 2025

0.175

Science Council of Japan foresight-0.896

Firms

0.601***

Government-0.103

Not for profit

0.761**

Age -40-0.520**0.158

Age 40-49-0.324*

0.528**

Age 60+-0.212

0.402

Member

0.153-0.901**

IT

0.085

Life science

0.551***

Nano/material

1.01***

Environment-0.066

Manufacturing

0.138

Frontier

0.068

Infrastructure

0.341

Energy-0.106

Follow the situation

1.88***

Cut point 1 (weak⇒moderately weak)-1.13

Cut point 2（⇒neither weak nor strong）

0.564

Cut point 3（⇒moderately strong）

2.690

Cut point 4（⇒strong）

6.41

Number of observation

552

544

77.39*** (dof=20)

9.65** (dof=4)

Hosmer-Lemeshow chi squared

-

0.17

Prob>Hosmer-Lemeshow chi squared

-

0.982

likelihood ratio chi squared

Impactadequate?

Note: Asterisks indicate significance at 10% (*), 5% (**), 1% (***). Results of “impact” showcoefficients of logistic regression with ordered category which affect the response according to
cut points. “follow” variables take the value 0 (not at all), 0.25 (not very often), 0.5 (neither), 0.75
(moderately often), or1 (often).

44

Table 15: perception on size of impact, divided by 1) panel member of not (row), and 2) how
often to follow the situation looking for any impact (column)
Not at all
PanelmemberNon-panel
memberN
Mean95%
confidence
interval of
mean
N
Mean
95%
confidence
interval ofmean

Not very
often

Neither

Often

Very often

Total

2

27

14

9

7

59

2.50

2.67

2.93

2.56

3.14

2.76

1.52-3.48

2.34-3.00

2.61-3.25

1.75-3.36

2.35-3.93

2.53-2.99

69

188

139

89

8

493

2.07

2.48

2.83

2.83

2.50

2.59

1.83-2.31

2.36-2.60

2.70-2.97

2.65-3.02

1.33-3.67

2.51-2.67

Note: The “mean” is calculated by taking average of “Weak” (=1), “Moderately weak” (=2), “Neither
weak nor strong” (=3), “Moderately strong” (=4) and “Strong” (=5). 95% confidence interval of meanis calculated by mean plus minus 1.96 x standard error of the mean.
Source: Survey conducted by the author

45