ISSN 1346-6151

Japanese Journal of
Evaluation Studies
Vol. 10, No. 2, September 2010
English Version

CONTENTS


Applicability of Empowerment Evaluation Approach in Community Program:
Implications from its Theory and Practice
Yuriko Minamoto 



Evaluation Method of International Cooperation on Climate Change
Kiyotaka Nakashima

Japan Evaluation Society

Japanese Journal of Evaluation Studies
Editorial Board
Koichi MIYOSHI

Editor-in-chief

Ritsumeikan Asia Pacific University

Keiko NISHINO

Vice-Editor-in-chief

GLM InstituteStanding Editors

Hiromitsu MUTA
Tokyo Institute of Technology

Editors

Atsuko AOYAMA
Nagoya University

Iwao OSHIMA
Japan College of Social Work

Tatsuya ONO

Kiyoshi YAMAYA
Doshisha University

Mari OSAWAThe University of Tokyo

Yoshiaki OKAMOTO
Mitsubishi UFJ Research & Consulting

Yoshio KUBOTA

Tottori University

Ryo SASAKI

Kobe Gakuin University

Mariko SATO

International Development Center of Japan

Kazuhisa SHIBUYA
Ministry of Land, Infrastructure,
Transport and Tourism

Yayoi TANAKA
National Institution for Academic
Degrees and University Evaluation

University of Tsukuba

Itoko SUZUKI
Kobe Women’s University

Shunji MATSUOKA
Waseda University

Yasuko MURAMATSU
Tokyo Woman’s Christian University

OfficeThe Japan Evaluation Society
c/o International Development Center of Japan
Hitachi Soft Tower B 22nd Floor, 4-12-6, Higashi-Shinagawa,Shinagawa, Tokyo, 140-0002, Japan
Phone: +81-3-6718-5931, Facsimile: +81-3-6718-1651
E-mail: jes.info@evaluationjp.org

Japanese Journal of Evaluation Studies is published twice a year in Japaneseand once or twice a year in English.
Japan Evaluation Society

Japanese Journal of Evaluation Studies
Vol. 10, No. 2, September 2010
English Version

CONTENTS
ArticleYuriko Minamoto
Applicability of Empowerment Evaluation Approach in Community Program:
Implications from its Theory and Practice …………………………………………………………1
Research Note
Kiyotaka Nakashima
Evaluation Method of International Cooperation on Climate Change ………………………………15
Publication Policy of the Japanese Journal of Evaluation Studies ………………………………………27
Information for Contributors (For English Papers) ………………………………………………………29
Writing Manual of the Japanese Journal of Evaluation Studies(For English Papers) ……………………31
Referee-Reading Guideline ………………………………………………………………………………33

1【Article】

Applicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice
Yuriko Minamoto
Meiji University
minamoto@kisc.meiji.ac.jp

Abstract
Empowerment evaluation advocated by David. M Fetterman is the use of evaluation concepts and techniques to fostertransformation and self-determination. In this article, applicability and issues of empowerment evaluation isinvestigated through case study on empowerment evaluation of community program for health promotion in Tokyo.
It provides an opportunity for a group of participants to empower themselves through evaluation activities as well as
to internalize self-evaluation mechanism for continuous improvement in the related organization. Major roles of
evaluators include facilitator, trainer of evaluation know-how and even direct advocate to help the group be
empowered. Empowerment evaluation appears most appropriate when the program itself has agenda ofempowerment and social transformation. The mixed use of empowerment evaluation and conventional evaluations
would make it possible to implement value-pluralistic evaluations. The accumulation of experiences of empowerment
evaluation in Japan is expected for further testing and adjustment of its theory and methodology.

Keywords
Empowerment, Participatory Evaluation, Process Evaluation, Transformation, Health Promotion

1. Introduction
In recent years, the need for participatory evaluation involving evaluation of policies from the perspective of service
recipients (beneficiaries) has become widely acknowledged in Japan. According to Furukawa (2000), behind thisphenomenon is the changing relationship of government and the public, as well as the rising prominence ofgovernance of the public sector including public interest activities by NPOs (non-profit organizations) and citizens.
And as the legitimacy of the public sector is scrutinized in its validity and legality, and further in the substantial
effectiveness of its policies and measures, engagement of the public in the evaluation of such aspects is expected to,strengthen the legitimacy of the government or its policies. A government s internal assessment is purely selfevaluatory; therefore, it becomes vital to conduct an evaluation by citizens as clients or partners of government,
services. Methods of participation in such evaluation vary- internal evaluation with reflection of citizens opinions ;Japanese Journal of Evaluation Studies, Vol.10, No.2, 2010, pp.1-13 [Revised based on the translation from Japanese
Journal of Evaluation Studies Vol.3, No.2, pp.70-86 (Japanese Version) and by adding a new case study. ]2Yuriko Minamoto

evaluation conducted in collaboration with such organizations as NPOs ; evaluation conducted by citizens
independent from a government (Nishio 2000, pp.30-31) among others-and there have already been cases that have,been put into practice. It goes without saying that the citizens capacity to take part in such assessment is imperativein evaluations conducted entirely by citizens. Yamaya (2000) finds the significance of empowerment evaluation,
which helps citizens self-develop their decision-making capacity, in having the potential to accomplish more.
If we turn our focus to the field of international development, participatory evaluation is regarded as a method
to enhance the quality of evaluations, as well as to increase the possibility of engaging key stakeholders in the,evaluation process allowing voices of people that exist on a different plane from the donors logic to be heard. Actual
cases of participatory evaluation are accumulating centering on programs implemented by Japan International
Cooperation (JICA, 2001). Additionally, there is much anticipation that participation of key stakeholders in theevaluation process will contribute to their heightened sense of ownership and assurance of autonomy, as well as a
new approach to evaluation that will bring about social transformation by implementing an evaluation process(Miyoshi & Tanaka 2001). Participatory evaluation is the participation and independent engagement of key parties
in the development process and the empowerment of the citizens in the process is a major factor. It is anticipatedthat learning and transformation is brought about by the process through citizen engagement in all program cycles asplanning, implementation, and evaluation. Though the term empowerment evaluation has not been adopted in thedevelopment assistance field, various research methods to assist the empowerment of people have been utilized inparticipatory development and social development programs1.
Amid heightening awareness for the need for citizens and key parties to engage in participatory evaluation,
empowerment evaluation is attracting attention as one of its methods. This article will consider its applicability,
focusing on the leading methodology of empowerment evaluation propounded by David M. Fetterman. First, it will,
sort out Fetterman s concepts of empowerment evaluation, followed by case study of the community program for
health promotion in Tokyo, and lastly, it will draw some implications on the application of empowerment evaluation.

2. Understanding Empowerment Evaluation from Program Theory Perspectives
Empowerment evaluation advocated by Fetterman (1996, 2001) is defined as providing people (group of people),
self-motivated in their attempt to reform their situation, the process to acquire self-determination through selfevaluation and reflection . Empowerment evaluation applies to both the empowerment process as well as its
outcome. For instance, empowerment process is the process in which people learn how to acquire self-determination,and consequently acquire the ability to seek ways to improve their situation. On the other hand, the result of theempowerment process (outcome) points to the group of people starting a movement having relative impact in theircommunity, or the actual impact as a result of the activity. This framework can be also discussed in the light ofempowerment theories for community empowerment or development that include both processes and outcomes forindividuals and organizations respectively. Empowerment processes for individuals might include participation incommunity organizations, whereas empowered outcomes for individuals might refer to situation-specific perceived
control and resource mobilization skills (Perkins & Zimmerman 1995, p.570). At the organizational level,
empowering processes deal with collective decision making and outcomes might include enhancement oforganizational network or organization itself. Applying this framework to program theory of interventions suggeststhat empowerment evaluation can be a part of activities or interventions to promote empowering process to bringabout empowered outcome and impact (Figure 1).
Another key feature of empowerment evaluation is the point that those participating in empowerment

Applicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice3

Figure 1 Empowerment Evaluation in Program Theory of Community Development

Impact in the Community

Empowered Outcomes
Individual level: Control and resource mobilization skills
Organizational level: Enhancement of network/ organizational capacity
Empowerment
Evaluation
Empowering Processes (Inputs, Activities,Outputs)
Individual level: Participation in community organizations
Organizational level: Collective decision making and shared leadership
Source: Author

evaluation are implementers, collaborators, and service beneficiaries, and their empowerment signifies theirconsequential development of strength as one group of key players . It seems there is an assumption that among the
key players, those involved in the intervention directly are self-motivated and hold a will to bring about some sort ofchange. In other words, a precondition for empowerment evaluation is that the intervention itself must possess anagenda for empowerment (Patton 1997, pp.156-157).

3. Features of Empowerment Evaluation Process
3.1 The Process of Dialogue
As with other approaches of participatory evaluation, empowerment evaluation, in which an evaluator functions as a
facilitator, takes an approach that places importance on the process of dialogue. Although it employs bothquantitative and qualitative evaluation methods, it begins by respecting the diversity of values under the assumptionthat facts are constructed . In other words, evaluation is not about seeking something that exists as facts , buttakes a position of constructivism, which follows the fourth generation evaluation method, where all parties involved
create a satisfactory situation through a negotiation process between evaluators and stakeholders (Guba & Lincoln
1989, p.11). It is therefore essential for evaluators and participants to follow a process in which they evaluate
collaboratively through dialogue.
Furthermore, in a dialogue-oriented process people are treated as equals in relationships. Standing on equal
footing means equality under diversity, signifying collaboration under the objective of evaluating a program to
improve social conditions, and that evaluation participants never become subjects of research or an experiment(Guba & Lincoln, 1989, p.11). As Hardina (2002) points out, the knowledge of experts in conventional approaches to4

Yuriko Minamoto

evaluation possess a high risk of further strengthening the traditional power structure between the haves and the havenots.
3.2 Five Keywords
Fetterman uses five keywords in explaining the concept of empowerment evaluation: (1) training of participants; (2)facilitation; (3) advocacy; (4) illumination and (5) liberation (Fetterman, 1996, pp.9-18). The first keyword training ofparticipants explicates that participants must be trained by evaluators prior to their evaluation, as participants later
conduct their own evaluation utilizing the empowerment evaluation method. This training internalizes the evaluation
into the program. In conventional evaluation approaches, evaluators leave as soon as the evaluation process ends;
however, empowerment evaluation stipulates that it is crucial for those involved in the program continue theevaluation process on their own. For the second keyword facilitation, Fetterman clearly states on the basis of his
conviction that people themselves can only empower themselves, and that the roles of external experts in theevaluation process are facilitators or coaches. Their specific task in the evaluation process is on a case-by-case basis,but their basic function is to offer counsel to the participants and a critical friend.
The third keyword advocacy stipulates that, in addition to the role as facilitators evaluators also play a role of adirect advocate to promote the empowerment of the participants (on which the empowerment evaluation centers)
through evaluation. An example would be in the case of the homeless, who cannot necessarily articulate theirsituation in their own words appropriately. The evaluators would suggest methods and direction to resolve theirproblems, and lead the way to social transformation. But involvement of evaluators in the advocacy of the program
itself is a contentious matter 2. Further, empowerment evaluation possesses features of illumination and liberation.
Empowerment evaluation encourages realization and enlightenment of all the parties involved through the evaluation
process. However, not only does it involve general realization and capacity building, but more specifically, it entails
becoming aware of the problem, building the skills to work out improvement measures, new strategies, and skills tomanage the program. Furthermore, empowerment evaluation brings about liberation through illumination. Liberationis the state in which those involved are freed from the oppressed condition or old notions and systems. This pertains
to the definition of empowerment, and can be thought as one of the prerequisites for self-determination.
3.3 Social Transformation
In empowerment evaluation theory, Fetterman claims that the idea of empowerment evaluation came to his mind
while writing his book Speaking the Language of Power: Communication, Collaboration, and Advocacy (1993). He
writes about his state of mind at the time as follows:
I wanted to explore the many ways that evaluators and social scientists could give voice to the people theywork with and bring their concerns to policy brokers. I found that, increasingly, socially concerned scholars
in myriad fields are making their insights and findings available to decision makers. These scholars andpractitioners address a host of significant issues, including conflict resolution, the drop-out problem,
environmental health and safety, homelessness, educational reform, AIDS, American Indian concerns, andthe education of gifted children. The aim of these scholars and practitioners was to explore successful
strategies, share lessons learned, and enhance their ability to communicate with an educated citizenry andpowerful policy-making bodies. Collaboration, participation, and empowerment emerged as common
threads throughout the work and helped to crystallize the concept of empowerment evaluation (Fetterman
1996, pp.6-7).

Applicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice5

As one can gather from the statement, empowerment evaluation has its roots strongly in social activism, and isclearly aware of social transformation positioned as a continuum of evaluation. Fetterman stands by his convictionthat only the parties involved can empower themselves, and the role of the external evaluators are supporters assisting
the disempowered to empower themselves; further, he supposes that they also play the role of an advocate for theparties involved.

4. Case Study of Empowerment Evaluation: Shinagawa Community Program for Health
Promotion (Kenko-Daigaku Shinagawa) 3
In this section, case of empowerment evaluation in one of local governments in Tokyo is studied to explore how thetheory can be applied to actual implementation of evaluation activities. The author has been involved in one training
session and two workshops as an evaluator and at the same time conducted interviews to the local government staff aswell as a post questionnaire survey to grasp the changing perceptions of participants.
4.1 Case Overview
Shinagawa Ward is located in Tokyo holding 349,829 populations (as of April, 2010). In fiscal year 2009, Shinagawa
Ward Office started a Shinagawa Community Program for Health Promotion (Shinagawa Kenko-Daigaku;hereinafter referred to as the Program) with objective of promoting health through community activities by people.
Such an idea well known as health promotion can be characterized as the process of enabling people to increase
control over, and to improve, their health , and it moves beyond a focus on individual behavior or curative healthtowards a wide range of social and environmental interventions 4. The Program intended to provide residents with
opportunities to become leaders or active members in health promotion activities in the community. In this sense, theProgram is associated with agenda of empowering people in the community.
Fifty residents have participated for 6 months course. According to interviews with the health administrative
staff involved in the Program, their motivation diversified from promotion of individual health condition by physicalexercise to enhancement of their capacity to deal with health issues in the community. The officers expressed concernthat their intention to promote health promotion activities at the community level might not be well understood by theparticipants. The officers actually were planning to formulate implementation committee with voluntarily members
from participants, so that the contents of the Program can reflect real needs of the community and residents
themselves can take initiatives. After the 1st year program terminated, the administration office decided to implement
participatory evaluation for improvement of the Program and enhancement of motivation of participants. They felt
the conventional evaluation by internal officer was not quite useful for those particular purposes. Moreover, top-down
approach to the community to provide the Program is somehow contradictory to diffuse health promotion ideas,
where initiatives and commitment of people and community groups are extremely important.
4.2 Empowerment Evaluation Procedure
The participatory evaluation for the Program, in essence similar to empowerment evaluation, was conducted after the
1st year program terminated. Before evaluation workshops, a training session is conducted to provide basic ideas andmethods of the evaluation. The objectives, main steps and process of the evaluation workshop are shared amongprospected participants. Providing training to participants is one of key characteristics of participatory evaluation
including empowerment evaluation, and an evaluator is expected to function as a trainer. In the following workshop
sessions, nineteen people have participated including head of the administration office (1), health administrative staff

6

Yuriko Minamoto(7), and residents (11) of Shinagawa Ward who finished the Program. Three steps were implemented in two,
workshop sessions with the evaluator as a facilitator (Figure 2). The first step starts with defining the Program s
goals, objectives and strategies using logic model concept. It was found that participants had a slightly different
expectation to the Program, therefore their perceived goals and objectives varied among them. Especially, difference
between administrators and residents become a critical discussion point. This step can be seen as a process forparticipants becoming conscious of program objectives as well as current health issues in the community.
The second step was to first identify significant concerns about strategy and activities of the Program in thelight of agreed objectives. They developed several evaluation questions which they themselves would like to know
their answers through evaluation. Those questions include relevance of the Program objectives and curriculum
development, and effectiveness of activities for respective objectives. Then, participants tried to answer thosequestions based on their observations and experiences in the Program. Rather than utilizing rating or voting system in
the workshop, various anecdotes and reasoning were open for the discussion. The significant point of this task is tocontinue a dialogue, crystallize the content and identify critical elements to be considered in future planning.
The third step involves the questioning the participants on how they want to pursue the activities given thecurrent situation. The participants will consider objectives and specific strategies identified in the previous steps. By
this time, all the participants, regardless their different stake, are appeared to look at same direction of the program
goals of health promotion as the results of dialogue process. After this evaluation practice, those involved in theworkshops are reported to become active members of the implementation committee for the Program in the following
fiscal year.
,
4.3 Review of Participants Perceptions
As discussed in section 2, empowerment evaluations can be defined as a part of intervention activities to promoteempowering processes of individuals or organizations and bring about empowered outcomes. In order to examine
whether the empowerment evaluation can contribute to expected processes or outcomes, the author has utilized ten
principles of empowerment evaluation to study the case. Fetterman and Wandersman (2005) set up empowerment
evaluation principles to clarify its concept and theory which often has been criticized by its ambiguity (Smith 2007).
Those principles include (1) Improvement; (2) Community ownership; (3) Inclusion; (4) Democratic participation,(5) Social Justice; (6) Community knowledge, (7) Evidence-based strategies, (8) Capacity building, (9)Organizational learning and (10) Accountability. Considering that the evaluated program is still at the 1st stage ofwhole intervention of health promotion policy in Shinagawa Ward and mainly focusing on empowering process, five

Figure 2 Empowerment Evaluation Procedure in the Case of Shinagawa Community Program

Training session
on empowerment
evaluation-fundamentals-procedures-objectives

1st workshop session

2nd workshop session-define program logic-identify concerns andissues in the program-develop evaluationquestions-discuss answers for
those questions

Facilitation, dialogue, inclusion, empowering process
Source: Author

Applicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice7,
principles among ten are used to examine the participants changing perceptions. Principles of improvement,
community ownership and community knowledge after the workshop, and democratic participation and inclusion are
inquired through questionnaire survey (Table 1). Other than these structured questions, open questions to selfevaluate their changes before and after the workshop are included.
4.4 Results and Discussions
The first principle of improvement reconfirms the practical aspect of empowerment evaluation that should not be
conducted for the sake of intellectual interaction alone. The objectives of the case evaluation are definitely set for
improvement, and social agenda of health promotion through community activities were clearly identified. The
questionnaire results indicated that four principles are positively perceived by the participants with mean score being
over 4.00 in 5-point scale (Table 1). Many of participants think problems and issues are clearly identified andmeasures for improvement can be fed back to the next phase. In the workshop, the residents who finished the 1st year
program were expected to be involved in program design and implementation and they were the ones who held
valuable information and knowledge about the social issues that are quite contextual in their community (principles
of community ownership and community knowledge). There were possibility for all the participants to mobilize own
knowledge for self-improvement, that can be extraordinary catalyst for change in the program group (Fetterman
2005, p.47).
The evaluator also played a meaningful role in democratic participation, where diverse stakeholders sit at thesame table to evaluate the program (principle of inclusion). According to the questionnaire survey, they highly
evaluated the workshop process in items of good atmosphere to express own opinions and well function ofevaluator to facilitate those discussions . The principles of inclusion and democratic participation are basics of
empowerment evaluation, where decision making power in evaluation design and interpretation lies withparticipating people. According to Fetterman (2005), empowerment evaluation helps organizations develop both theclimate and structure for generating reflective practitioners (p.50). After the workshop, the health administrators andresidents who participated in the workshop together have been engaged in a new program development and thoseresidents also become the main actors to support new participants in the program. This workshop entailed the strong
intention to locate this process in whole management cycle of the Program. According to the interview to the healthadministrator a month after the workshop, the new program has been developed together with those participants andit clearly presented the mission, objectives and strategy based on what has been discussed in the workshop.
When we compare degree of changes of perceptions between the health administrators and residents, the healthadministrators rated higher in all aspects (p<.01). This may give us notion that the empowerment evaluation was
introduced with strong leadership of health administrators, and residents remained passive participants in theworkshops. It also implies that there may be some gaps between empowering processes and empowered outcomes ofpeople being proactive actors in the community. Continuous use of empowerment evaluation in different settings ofthe program operating cycle should be carefully considered in order to contribute to the achievement of empowered
outcomes.
Empowerment evaluation methodology consists of a series of processes that include affirmation of the program
mission through debate among participants, evaluation of the present situation, and drawing up of a future plan fromthe results. In empowerment evaluation, it is suggested that the repetition of the above-mentioned processes is
crucial; a break in the process cannot lead to the fostering of self-determination (Fetterman 2001, p.13). As illustrated
in the case example, empowerment evaluation is not a type of an evaluation research where evaluation results are
drawn in a specific time period, but its key feature is its continual evaluation process, which takes place over an
extended period of time. In addition to these key methodological features, empowerment evaluation, which includes

8

Yuriko Minamoto

Table 1 Five Principles and List of Statement in Questionnaire Survey

Principles

Statements (5-point scale)●

Improvement(mean=4.05)

●
●
●
●

Community ownership(mean=4.04)●●●

Community knowledge(mean=4.26)
Democratic
participation/inclusion(mean=4.56)

●
●
●

●
●

Problems and issues are clearly identified.
Objectives are clarified.
Measures for improvement are fully discussed
Measures for improvement can be fed back to the next phase.
Commitment to the Program are shared among participants
Degree of involvement in the Program is clarified
Willingness to be involved in the Program is heightened.
Sense of inclusiveness among participants is strengthened.
Various ideas are discussed.
Understanding of Program content is enhanced.
Leaning from other participants is promoted.
Good atmosphere for active participation is secured through workshop.
Facilitator (evaluator) functions well to promote participation

Source: Authortransformation, capacity building and community ownership, requires understanding of its philosophy-only use of its
tools itself is inadequate.
Moreover, as seen in the case example, the formation of a group comprised of those involved in the community(for the case example, the implementation committee of the Program) is critical in empowerment evaluation to
become proactive actors of the intervention. It is crucial for the Program that both administration staff and residents
take on a sense of involvement as one group . That is, there must be a process of transformation in which those who
might have been a third-party (participants of the Program, people not having shown any interest to healthpromotion, etc.) up to that point change into an actor playing some sort of role in the Program. The biggest featureof empowerment evaluation is such broadening of parties involved .

5. Implications on the Application of Empowerment Evaluation
This section will present the types of interventions suited for empowerment evaluation, as well as points to consider,in light of the definition, methodology, and case example aforementioned in this article.
5.1 Potentiality and Scope of Application
Firstly, empowerment evaluation is suited to an evaluation of intervention that itself contains an empowerment
process of participants. Further, in terms of identifying participants as self-empowering people , the scope ofparticipants will evidently be small, rather than those involving a large unidentified number of citizens. Suchexamples would include civic activities, public social programs, as well as those whose objective is to transform asituation within a community. Because empowerment evaluation is a continuous evaluation process, using theevaluated program as an entry point, such process that promotes the empowerment of people leading to theApplicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice9internalization of the evaluation activity may well become a tool to facilitate the broadening of the program impact
expected in the community.
Secondly, in terms of programs carried out in Japan, empowerment evaluation may be utilized to evaluatesocial service programs by NPOs (including those contracted out from the government). NPOs operate on the basis
not of economic values but of other diverse values, with voluntary participation and support from citizens.
Additionally, many NPOs have a role in promoting social transformation operating on such key ideas as selfmanagement and self-determination (Amenomori 2002, pp.24-28). The employment of empowerment evaluation in
the activities of NPOs working especially in such areas as public welfare (of the disabled and the elderly), regional
development, and rural development (in isolated and depopulated areas) that affect socially disadvantaged people
would carry a significant meaning because the ultimate goal of these activities is to support the autonomy of thesepeople who are also service beneficiaries. Such example would be a NPO providing elderly care services. To question
the mission of the program in empowerment evaluation signifies the need to establish what autonomy of the elderlymeans, and thus it becomes imperative to listen to the service users. Not only is it necessary for the service
beneficiaries to participate, but also their family, caregivers who provide the service, and community volunteers need
to be involved. All participants will take stock keeping in mind what kind of service provided (or what kind ofservice selected, from the perspective of the service users) will make autonomy closer to reality, and then form afuture strategy. It would be ideal for the elderly themselves to take a step in the direction towards their autonomy
during this process. Further, it is anticipated that community members will become involved in issues concerning theelderly, as well as in the operation of the NPO as central players in the community, and not as a third-party
bystanders. The dialogue process, on which empowerment evaluation places emphasis, will provide a momentum toreexamine the power structure between service providers and service users (the disadvantaged) in social services(Sasagawa Peace Foundation 2001, p.19). In programs implemented collaboratively with the government,
empowerment evaluation can be utilized as a tool to construct a public will reflecting the diverse values of its
citizens. This will carry a substantial meaning in sharing objectives, which is the basis of running collaborative
programs.
5.2 Challenges and Points to Consider
One of the challenges in conducting the evaluations discussed is the awareness and skills of the evaluator. The needfor the evaluator to function as a facilitator or a coach instead of an assessor has already been discussed earlier, but it
must be noted that a different set of roles and skills from when conducting conventional methods of evaluation,accompanies the role. The evaluator must first hand down one s expert knowledge to the participants, and then must
support them through training to become capable in evaluating using that knowledge. The evaluator is in a position tofoster new evaluators. Additionally, one must have the ability to conduct workshops from a neutral standpoint (inother words, have the capacity to relativize diverse opinions) and further have the skills to devise measures for allparticipants to exchange opinions on an equal footing. The final objective of the facilitation process is not necessarily
the establishment of a consensus among the participants, but rather the creation of an interaction among people
through dialogue. Communication that develops during the evaluation process shakes off bias, and in empowerment
evaluation, in which process self-accountability is achieved, the role of the evaluator is significant. And above all, it
is vital for the evaluator to understand the social transformation and the direction of empowerment that the evaluatedprogram is aiming for. Conceptually, this means that the evaluator himself/herself are put right in the middle of the
program, and as an expert they are to support the evaluation in the program as well as the program that has
internalized the evaluation. One of the biggest features of empowerment evaluation is, perhaps, such new roles given
to the evaluator.

10

Yuriko Minamoto

There are 4 points to consider when contemplating the role of the evaluator and his/her relationship to
empowerment evaluation. First is the issue of objectivity. In empowerment evaluation, as represented byconstructivism in the fourth generation evaluation, the outcome of the evaluation does not exist as a fact but issomething that is constructed during the interactive process between all involved in the evaluation, including theevaluator, and thus can be regarded as not considering the existence of an objective party to begin with. It takes the
position that evaluation outcomes are produced as a result of diverse values colliding with each other, and it is not theright answer that can be scientifically verified, but it should be treated as something that will continue to benegotiated among those involved even after the evaluation ends. That is, there is no need for a premise foruniversalization. Continuation of negotiation following the end of evaluation signifies a repeated process brought
about by internalization of the evaluation.
Second is the problem of bias. In empowerment evaluation, it stands on the basis that bias will be removed bymutual oversight of the participants through the process of dialogue. In other words, it presupposes that, by advancing
communication among participants on an equal footing during the evaluation process, their commitment is increased
and thus reliable debate will unfold (Fetterman, 1996, p.23). Moreover, it hypothesizes that it is possible to achieve,
mutual understanding by properly laying out one s reasoning during the debate. However, a question remains whether
such ideal communication, under the presumption that people can be rational through dialogue, is consistently
possible. A situation where contrasting interests conflict sharply is easily conceivable. Furthermore, there may be bias
in the selection of participants, as such case as only those with the motivation to reach consensus could participate in
the evaluation. Nevertheless, the process of defining the mission and goals at the beginning of empowerment
evaluation can be seen as a powerful tool to reduce bias in the sense of redefining the motivation in forming a
consensus.
,
Third is the issue of the evaluator s neutrality by becoming an advocate of the program. Patton (1997) points outthat it is not unusual for the evaluator to advocate the evaluation outcome in participatory evaluation; however, it
becomes a contentious matter when empowerment evaluation takes a step further by including advocacy of theevaluated program itself (pp.157-159). By becoming an advocate for the evaluated program, there is danger of the,evaluator s neutrality being distorted, destroying the reliability of the evaluation outcome. In internal evaluations, theevaluator proactively advocating the evaluation outcome is acknowledged as a role to promote program improvement
as well as organization improvement. The internal evaluation is recognized as an impartial action committed after theevaluation process, and hence does not damage the reliability of the evaluation (Sonnichsen, 2000). Fetterman
himself emphasizes that, also in the case of empowerment evaluation the evaluator only advocates the program
following the evaluation outcome and should not become committed to the program regardless of the evaluation
results; however, at the same time, he assumes a situation where the evaluator takes on the role of supporting asocially disadvantaged group that cannot empower themselves (Fetterman 1996, pp.12-13; Fetterman 2001, pp.115117). Empowerment evaluation differs from conventional internal evaluations in that those involved in the program
themselves provide the process to enable their empowerment, and moreover, there exists a mechanism where they,
themselves conduct the evaluation in which their empowerment and transformation become possible. It is the authors
notion that in order for that process to function effectively the evaluator involved in the process must comprehend theconcept of empowerment as well as be ready, to some extent, for a long-term commitment to the program itself. Thatis perhaps the reason behind calling on an outside evaluator to take on the role of a social activist. However, in reallife application, strong caution must be taken so that the evaluator does not become overly involved in the program tothe point of manipulating the facilitation process.
Fourth point to take into consideration is the fact that empowerment evaluation is only one way ofstrengthening self-determination and to bring about some sort of transformation. As observed in the case example,

Applicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice11

there seems to be gaps between empowering process and empowered outcomes of the Program. The mechanism inwhich the enhancement of self-determination leads to social transformation is unknown. Further, to achieve the goalsof the program as a whole, there must be an administrative structure or institutional arrangement in which room is left
for decision-making by the group of people affected. Perceived from a systematic view, empowerment evaluation
sheds light on a process that forms an objective significance that can be shared among those involved. There will be aneed to further research concerning the mechanism of the relationship between empowerment evaluation andorganizational and institutional enhancement.

6. Conclusion,
Empowerment evaluation, in Fetterman s words is the use of evaluation concepts, techniques, and findings to foster
improvement and self-determination (Fetterman 1996, p.6). It is also an evaluation process that possesses new facets
as social transformation, emphasis on dialogue process, role of evaluators in advocacy, internalization of evaluation
activity, and liberation of participants. Empowerment evaluation is an evaluation approach that encompasses thephilosophy of empowerment and social transformation, conducted by the parties involved. That is, when seen from adifferent perspective, it is not an evaluation approach in which the parties involved participate in the evaluation,but one in which the evaluator participates as a facilitator or a coach, in the independent program implemented by
the parties involved.
The case study indicates that empowerment evaluation approach can provide the process for a group ofparticipants concerned to empower themselves in terms of gaining knowledge and ownership and improving thefuture program. The theory is applicable especially to a program with empowerment agenda. Evaluations differ in its
applicable theory and method according to their objectives and characteristics of evaluation object; therefore, as
Fetterman himself states empowerment evaluation does not exclude other conventional approaches of evaluation. At
the same time as the case program carries out an empowerment evaluation with service users and parties involved, an
impact evaluation to examine attribution of a given intervention to outcomes will be necessary. In a sense, for those
conducting a conventional approach of evaluation, identification of program framework will already be done by theparticipatory approach responding to their own logic 5. The combination of empowerment evaluation and conventional
evaluation as a management tool is a noteworthy form of evaluation that brings in a pluralistic perspective of society,
Fetterman s empowerment evaluation is a method that ties in evaluation techniques with empowerment andsocial transformation. Furthermore, it questions how evaluation can benefit the autonomy of the vulnerable groups.
Needless to say that empowerment evaluation alone cannot achieve such autonomy. By accumulating more cases of
empowerment evaluation, it will become necessary to examine its theory and methodology, its relation to socialtransformation and autonomy, as well as the mechanism of organization and program improvement. That task may
become a catalyst to consider how public programs and interventions with agenda of empowerment could be
effectively implemented for public welfare.

Notes
1 Examples include PRA (Participatory Rural Appraisal) and PLA (Participatory Learning and Action), among others. However,it has been pointed out that in their application, there are many cases in which only their tools are utilized, and conducted as aparticipatory process but in reality ending in a superficial participation of the citizens that does not include their information
input or allow their decision-making (Sato, 2003).

12

Yuriko Minamoto

2 For example, Patton (1997) throws out a question whether an evaluator who also acts as a program advocate can deliver
appropriate evaluation from a reliability and utility standpoint.
3 The author participated as an evaluator in the workshops. The author would like to extend sincere appreciation to Shinagawa
Health Office to provide the author with valuable experiences of empowerment evaluation activities.
4 WHO homepage (http://www.who.int/topics/health̲promotion/en/)
5 Confirmation of mission and objectives, as well as reviewing verifiable indicators in empowerment evaluation can also be
considered as a part of Evaluability Assessment.

References
Amenomori, T. (2002). NPO Ron. Aichi, Japan: Nihon Fukushi University Distance Learning.
Brisolara, S. (1998). The History of Participatory Evaluation and Current Debates in the Field. Understanding and Practicing

Participatory Evaluation: New Direction for Evaluation, 80, 25-41.
Cousin, J. B. and Whitmore, E. (1998). Framing Participatory Evaluation, Understanding and Practicing Participatory Evaluation:

New Direction for Evaluation, 80, 5-23.
Fetterman, D. M. (1996). Empowerment Evaluation-An Introduction to Theory and Practice, In D. M. Fetterman, S. Kaftarian, &A. Wandersman (Eds.), Empowerment Evaluation-Knowledge and Tools for Self-Assessment & Accountability (3-45).
Thousand Oakds, CA: Sage.
̲̲̲̲̲̲̲̲ (2001). Foundation of Empowerment Evaluation. Thousand Oaks, CA: Sage.
Fetterman, D.M and Wandersman, A. (2005) Empowerment Evaluation Principles in Practice, The Guilford Press.
Freedman, J. (1992). Empowerment-The Politics of Alternative Development. (C. Saito & T. Amenomori Trans.). Oxford,
England: Blackwell.
Furukawa, S. (2000). Shimin Sanka no Hyoka Hoshiki-Seidoka no Kanosei. In M. Nishio (Ed.), Gyosei Hyoka no Choryu-

Sankagata Hyoka Shisutemu no Kanosei (39-76). Tokyo: Institute of Administrative Management.
Guba, E.G. and Lincoln, Y. S. (1989). Fourth Generation Evaluation. Thousand Oaks, CA: Sage.
Hardina, D. (2002). Analytical Skills for Community Organization Practice, New York: Columbia University Press.
Inomata, M. (1992). Soshiki no Communication Ron. Tokyo: Chuo Keizai Sha.
Japan International Cooperation Agency. (2001). Participatory Evaluation and International Cooperation. Tokyo: JICA ResearchInstitute.
Levin, H. M. (1996). Empowerment Evaluation and Accelerated Schools, In D. M. Fetterman, S. Kaftarian, & A. Wandersman(Eds.), Empowerment Evaluation-Knowledge and Tools for Self-Assessment & Accountability (49-64). Thousand Oaks, CA:
Sage.
Miyoshi, K. and Tanaka, Y. (2001). Sankagata Hyoka no Shoraisei-Sankagata Hyoka no Gainen to Jissen ni tsuite no Ichi
Kosatsu. Nihon Hyoka Kenkyu, 1 (1), 65-79.
Nishio, M. (2000). Gyosei no Hyoka Hoshiki no Kakucho wo Mezashite. In M. Nishio (Ed.), Gyosei Hyoka no Choryu-

Sankagata Hyoka Shisutemu no Kanosei (1-35). Tokyo: Institute of Administrative Management.
Patton, M. Q. (1997). Toward Distinguishing Empowerment Evaluation and Placing It in A Larger Context. Evaluation Practice,
18 (2), 147-163.
Perkins, D. and Zimmerman, M.A. (1995). Empowerment Theory, Research, and Application, American Journal of Community

Psychology, 23(5)Saito, F. (Ed.). (2002). Sankagata Kaihatsu. Tokyo: Nihon Hyoron Sha.
Sasagawa Peace Foundation. (2001). Hyoka no Susume-Fukushi Bunya de Katsuyaku Suru Hieiri Soshiki No Tame Ni. Tokyo:
Sasagawa Peace Foundation.

Applicability of Empowerment Evaluation Approach inCommunity Program: Implications from its Theory and Practice

13

Sato, H. (Ed.). (2003). Re-thinking of Participatory Development. Chiba, Japan: Institute of Developing Economies.
Smith, N.L. (2007). Empowerment Evaluation as a Evaluation Ideology, American Journal of Evaluation, 28 (2), 169-178.
,
Sonnichsen, R.C. (2000). High Impact Internal Evaluation -A Practitioner s Guide to Evaluating and Consulting Inside

Organizations. Thousand Oakds, CA: Sage.
Yamaya, K. (2000). Hyoka no Tayosei to Shimin-Sankagata Hyoka no Kanosei. In M. Nishio (Ed.), Gyosei Hyoka no Choryu-

Sankagata Hyoka Shisutemu no Kanosei, (77-108). Tokyo: Institute of Administrative Management.
(Accepted 26th August, 2010)

15【Research Note】

Evaluation Method of International Cooperation on
Climate Change
Kiyotaka Nakashima
Environmental Leadership Development Project Coordinator of Iwate University
knakashi@iwate-u.ac.jp

Abstract
This paper considered how to evaluate international cooperation on climate change. Firstly, an evaluation framework
was established by the interdisciplinary review from the political, legal and economic dimensions of social science inenvironment policy. This is the framework that consisted of elements about the international cooperation
systematically. Secondly, this paper examined the effectiveness and subjects of the established evaluation framework.
,
By using this framework, the case-study showed the evaluation result and policy subjects of Japan s international
cooperation. The result of this case-study could verify the effectiveness of the framework, because international
cooperation was able to be evaluated synthetically. One subject is to increase the number of case-study, and the otheris to change the components of the framework. This paper could propose the qualitative framework as the evaluation
method of international cooperation on climate change by materializing two methodological strong points ofenvironmental policy, which are interdisciplinarity and synthesis.

Keywords
climate change, international cooperation, evaluation framework

1. Introduction
This paper considers how to evaluate international cooperation on climate change (CC) as environmental policy (EP)through positioning policy evaluation as one of the evaluation theories. Firstly, based on EP evaluation theory, whichis positioned as a discipline within the field of policy evaluation, an evaluation framework for the international
cooperation on CC is established based on the field of social science in EP. Next, this paper discusses theeffectiveness and research subjects of that framework through case studies on international cooperation1.
We recognized one turning point in global multilateral cooperation on CC in 2008. The year 2008 is the firstyear of the first commitment period of the Kyoto Protocol (KP).
In Article 3.1 of KP, all developed countries shall reduce their overall emissions of six greenhouse gases
including CO2 by at least 5% below the base year of 1990 or 1995 levels in the commitment period from 2008 to

Japanese Journal of Evaluation Studies , Vol.10, No.2, 2010, pp.15-26 [Translated from Japanese Journal of
Evaluation Studies, Vol.9, No.1, pp.19-29 (Japanese Version)]

16

Kiyotaka Nakashima

2012. In addition, each developed party has their own QELROs (Quantified Emission Limitation and Reduction
Objectives).
Now the question is how much preparation the developed countries have undertaken since the adoption of KP
in December 1997 toward the implementation of the first KP commitment starting from January 2008. It is possible
to identify the possibility of whether developed countries can implement their commitments through evaluating the
progress of this preparation. This evaluation is related to the direction of global multilateral cooperation to combat
CC.
Evaluation described in this paper is to verify the progress and process of the commitment implementationand international cooperation on CC undertaken by developed countries through establishing and applying theframework that includes evaluation standards.
Upon understanding that CC is a subject of social intervention2, this paper examines how policy responses are
implemented and their effectiveness. This is an important research theme within EP evaluation theory. Theinternational cooperation in this paper is evaluated from the perspective of the theory.
Next, we need to consider how we can evaluate commitment implementation and the progress and process ofinternational cooperation by developed countries.
This evaluation method should take the characteristics of CC into consideration.
CC is recognized as a common problem for human beings, because it arises from increasing the concentrationsof greenhouse gases discharged through human activities and has adverse effects on human beings on a global scale
over a long period of time. Also, CC arises from diverse, mutually intertwined factors.
To solve a problem with such characteristics, we need to proceed with the policies of mitigation to reducegreenhouse effect gases and those of adaptation to deal with the adverse effects of CC, simultaneously. The
commitments for mitigation and adaptation are imposed on developed countries by the UNFCCC (the United Nations
Framework Convention on Climate Change) and KP. It is essential to implement these commitments in order to solve
the CC problem.
This paper applies an EP approach with the methodological strengths of interdisciplinarity and synthesis
in order to evaluate the situation of commitment implementation regarding mitigation and adaptation by thedeveloped countries3.
Whereas the research in the field of EP on CC is experiencing specialization into natural and social sciences, italso shows trends of interdisciplinarity and synthesis as is observed in the integrated environmental assessment
model 4.
We can recognize this trend as the evaluation method that is based on the characteristics of CC, namely that it iscaused by various complex and mutually intertwined factors. This trend is an attempt to overcome the limitation of
specialization in the individual academic fields.
The contribution of economics, psychology, politics, and anthropology to the development of the evaluation
field demonstrates this, and is also connected to extensive interdisciplinary mutual borrowing. Thus the evaluation
field has an interdisciplinary character 5.
This paper tries to materialize the methodological strong points, interdisciplinarity and synthesis, as theanalytical framework to evaluate international cooperation on CC. This framework is established to evaluate theinternational cooperation of developed countries toward implementing the commitment of mitigation and adaptation.
The establishment of the framework is an attempt to shape the characteristics of interdisciplinarity andsynthesis in a way similar to the integrated environmental assessment model.
However, the framework in this paper is different from the integrated environmental assessment model, because
it tries to evaluate the international cooperation regarding mitigation and adaptation by developed countries at theEvaluation Method of International Cooperation on Climate Change

17level of project and program, based on the hierarchical structure of the policy 6.
,This paper evaluates Japan s cooperation with international efforts to combat CC through evaluating the
projects and programs conducted by the government of Japan. In this sense, it is a unique attempt in the field of
evaluation theory as it relates to climate policy.
The establishment and application of the framework is a specific approach to examine the evaluation method
for EP concerning CC. This paper aims to contribute to the methodological development in EP evaluation theory.
To give an overview of this paper, firstly, this paper establishes the framework by systematically connecting
multiple elements of international cooperation on CC.
Secondly, it examines the effectiveness and research subjects of the framework by applying it to the case study,of Japan s international cooperation.
Lastly, a summary of the result of examining the framework is given.

2. Establishing the evaluation framework
This section establishes the evaluation framework for international cooperation on CC and shows the content of theframework in Figure 1.
This framework consists of the following four elements (components).
Component (1)Simultaneity of the international negotiation process and the commitment implementation process
Component (2)Mutual supplementation between mitigation and adaptation on CC
Component (3)The relations between and interactions of domestic and international aspects
Component (4)Coordination among principle, commitment, commitment implementation measure, and between
evaluation standards
This paper classifies international cooperation on CC into the international negotiation process and thecommitment implementation process, as well as into mitigation and adaptation with regard to components (1) and (2).
International negotiation process is the process in which international agreements regarding CC are
determined. The commitment implementation process is the process in which policies and measures are carriedout to meet the commitment set forth in international agreements.
The policies and measures by the parties are classified as domestic political measures or international
cooperation. Adaptation assistance" is a term used to denote international cooperation on adaptation, and focuseson the assistance from developed countries to developing countries.
Components (1) and (2) connect the international negotiation process and commitment implementation process,as well as mitigation and adaptation, in order to evaluate the international cooperation on CC on the basis of priorstudies and the trends of international negotiation and EP.
Components (3) and (4) are drawn from the interdisciplinary review of prior studies regarding international
cooperation on CC from varying fields of social science relating to EP, including politics, law, and economics.
Component (3) is the interaction and relatedness of domestic and international aspects. This is based on the factthat prior studies in politics and law regarding EP discussed the interaction of domestic and international politics and18

Kiyotaka Nakashima

Figure 1 An evaluation framework for international cooperation on climate change（2）
Mutual supplementation between mitigation and adaptation on climate change
mitigation

adaptation

principle//

efficiency

equitycommitment/

elements of international cooperation
on climate change/ commitment implementation measure provisions of international agreements

evaluation standards ofinternational cooperation on CC

effectivenessinternational
negotiation（1）
Simultaneityoftheinternational
negotiation
processandthecommitment
implementation
process

supplementarity

domestic political measures/international cooperation

financial additionality

issues of international cooperation
on CCinternational cooperationcommitment implementation process

comprehensive climate policy
commitment
implementation
process

process

policyex:energy conservation program

ex:CDM program

ex:environmental ODA program

programex:energy conservation projectex:CDM project

ex:environmental ODA projectproject

developed country responsibility/

future commitment of developing coutries（4）
Coordination
among principle,commitment,
commitment
implementation
measure, andbetweenevaluation
standards（3）
The relations
between andinteractions ofdomestic andinternational
aspects

Factors obstructing
global multilateral cooperation

evaluating international cooperation and pointing out the policy subject on CC by the research subject country

Source: Nakashima (2007a)the relationship of domestic and international laws7.
Also, component (4) is the coordination among principle, commitment, and commitment implementation
measure, and between evaluation standards. This is based on the fact that prior studies in the legal and economic
fields concerning EP connected the evaluation standards for international cooperation, i.e. equity, efficiency, andeffectiveness, with the principle, commitment, and commitment implementation measure based on international
agreements on CC 8.
Principle indicates the guideline for the parties of UNFCCC and KP to implement the climate policy. Equity
as an evaluation standard is observed in the principle of equity and the principle of common but differentiatedresponsibilities specified in Article 3.1 of UNFCCC. Also, efficiency is included in the global cost-effectivenessregarding the precautionary principle in Article 3.3 of UNFCCC.
The commitment is divided into mitigation and adaptation assistance. The mitigation commitment is describedin Article 4.2 (a) of UNFCCC and Article 3.1 of KP. Also, the commitment of adaptation assistance is included in
Article 4.4 and 4.8 of UNFCCC. The effectiveness as an evaluation standard is the implementation of thesecommitments.
Prior legal and economic studies focusing on EP consider the effective enforcement (effectiveness) of thepolicy & measures (commitment implementation measure) for the implementation of the commitment on the basis ofequity and efficiency. These studies discuss the coordination of the principle and commitment in international
agreements as well as that of equity and efficiency, which are the evaluation standards for international cooperation
on CC.
Moreover, we can derive the relationship among the three evaluation standards from component (4). In otherwords, it is necessary to consider the influence of effectiveness caused by the conflict or coordination between equity
and efficiency to evaluate international cooperation on CC. This consideration is to inspect the coordination between
the principles and commitments in the international agreement on CC.
Next, this paper explains in detail the four internal aspects of the framework.
The inside of this framework in Figure 1 consists of some related elements in international cooperation on CC.
These components are the provisions of the international agreement on CC, the evaluation standards and the issues ofEvaluation Method of International Cooperation on Climate Change

19international cooperation, the system of measures for the implementation of commitments, and the factors obstructing
global multilateral cooperation.
Component (4) relates the principles and commitments of the international agreement to equity, efficiency, andeffectiveness, which are the evaluation standards.
The issues in international cooperation are associated with the provisions of international agreements and theevaluation standards of international cooperation on CC. These issues are what were discussed in the international
negotiation process. To follow up on this we investigate how the developed countries that are the subjects of thisresearch deal with issues in the commitment implementation process.
Figure 1 shows supplementarity and financial additionality as the issues of international cooperation.
Supplementarity indicates the positioning of the Kyoto mechanism, a means to enable international cooperation
on mitigation, as a supplement to domestic policy & measure. This paper regards this as a narrow definition anddefines supplementarity in a broad sense as the issue of coordination between domestic policy & measure andinternational cooperation concerning mitigation of CC.
Similarly, financial additionality is defined as no diversion of ODA (Official Development Assistance) for
CDM (Clean Development Mechanism). This paper regards it as a narrow definition, and defines financial
additionality in a broad sense as the issue of using official finance including ODA for Kyoto mechanisms such as
CDM. Financial additionality in a broad sense is considered in terms of international cooperation regarding
mitigation and adaptation.
The two issues are examined from the perspective of whether each developed party ensures them in thecommitment implementation measure.
The commitment implementation measures are classified into policies, programs and projects based on thehierarchical structure of the policy.
It is the comprehensive climate policy and plan that includes the domestic political measure and international,
cooperation on mitigation or adaptation. For example, Japan s comprehensive climate policy and plan is the
Outline for Promotion Effects to Prevent Global Warming and Kyoto Protocol Target Achievement Plan .
The programs and projects on mitigation and adaptation are carried out on the basis of these comprehensive
policies and plans. Program is looked upon as a set of projects here.
Figure 1 shows the energy conservation program and project as a domestic political measure on mitigation,
CDM program and project as international cooperation on mitigation, environmental ODA (governmental
development aid for environmental protection and conservation) program and project as adaptation assistance. This
paper examines whether developed countries ensure supplementarity and financial additionality through these
programs and projects.
The situation of securing the two issues is used to consider the influence on the obstructing factors for global
multilateral cooperation towards solving the CC problem.
,
This paper focuses on the developed countries responsibility and the future commitment of developing
countries as the factors obstructing global multilateral cooperation in Figure 1.
,
Developing countries insist on the Developed countries responsibility , a claim they make which situations
that developed countries must deal with the CC problem first because they caused it 9.
Similarly, developed countries insist on the future commitment of developing countries , advocating that thelegally binding QELROs also be imposed on developing countries or that developing countries should decide theirown QELROs.
These two obstructing factors are tied to the relationship between developed and developing countries when
discussing global multilateral cooperation.

20

Kiyotaka Nakashima

This paper evaluates the international cooperation and points out its policy subjects through considering theirinfluence on the two obstructing factors in the projects and programs undertaken by the country studied on the basisof the framework in Figure 1. This is the procedure used to evaluate the international cooperation on CC mitigation
and adaptation.
The next section explains this procedure in detail by using the applied example of the framework.

3. Effectiveness and research subjects of the evaluation framework
Section 2 established the evaluation framework for international cooperation on CC. This was set up based on thesituation of international negotiations, the content of international agreements as the result of the negotiations, and theresearch review concerning EP on CC.
This section discusses the effectiveness of the framework and points out its research subjects through using theapplied example.
3.1 Evaluation procedure for international cooperation and the application of the evaluation framework
This subsection gives an outline of the evaluation procedure for international cooperation on CC and the applied casestudy of the evaluation framework to discuss the effectiveness as well as the research subjects of the framework as
established in section 2.
As shown in Figure 1, the case study is the programs and projects of CDM as international cooperation
regarding mitigation by developed countries, and the environmental ODA regarding adaptation assistance undertakenby them.
This paper focuses on CDM and environmental ODA as the case study, because the point of these projects is
the relationship between developed and developing countries.
We regard CC as a common problem for human beings. Therefore, it is essential to establish the global
multilateral cooperation of developed and developing countries to solve the problem. This paper insists that we canunderstand the present status of cooperation between the developed and developing countries through evaluating the
projects and programs of CDM and environmental ODA involving developed countries. This will also result in
evaluating the progress of global multilateral cooperation on CC.
As the evaluation procedure, this paper first establishes the following four evaluation items to evaluate theinternational cooperation of developed countries by reviewing the international negotiation process on the basis of theframework10.
1）The policy equity esteem and efficiency limitation is seen in the international negotiation process.
Is this policy followed in the commitment implementation process too?
2）Does the contradiction of equity and efficiency have adverse effects on the effectiveness ofinternational cooperation on CC?
3）Is there a contradiction between the principles and commitments in the international agreement on
CC?
4）Can we see the influence of the obstructing factors on global multilateral cooperation, developed
countries responsibility and future commitment of developing countries ?
The projects and programs of CDM and environmental ODA in the commitment implementation process areevaluated on the standards of: equity, efficiency, and effectiveness. Moreover, the four evaluation items and how

Evaluation Method of International Cooperation on Climate Change

21each developed party ensures supplementarity and financial additionality are examined.
Based on the results of this case study, this paper evaluates the international cooperation of developed countries
that are the target of this research in regards to mitigation and adaptation, and points out the policy subjects.
Nakashima (2007a, 2007b)11 focused on Japan as the research country and evaluated the international
cooperation regarding mitigation and adaptation on CC.
The reason for taking Japan as an example is because Japan has an important responsibility and role ininternational cooperation on CC, while having some of the most serious domestic circumstances among thedeveloped countries with regard to the implementation of the mitigation commitment.
This paper evaluated the international cooperation up to 2007 based on the understanding that the situation isconstantly changing and because the year 2008 is the time to evaluate the implementation of the commitment by theJapanese government.
The case study focuses on the CDM project in Vietnam and the environmental ODA project in Maldives asboth states are developing countries in Asia closely related to Japan. In addition, it discusses the program of CDM
and environmental ODA as a set of projects undertaken by the Japanese government.
Judging from the result of the project and program evaluations of CDM and environmental ODA in the cases
studied, it was confirmed that the Japanese government is preparing for the procedure and plan to ensure
supplementarity and financial additionality in a narrow sense with regard to CDM and environmental ODA.
Also, as the result of examining the four evaluation items, no adverse effects on the effectiveness ofinternational cooperation deriving from the contradiction between equity and efficiency were found. In addition,compliance with the policy equity esteem and efficiency limitation was also observed in the commitment
implementation process as well as the international negotiation process.
Based on the result of this case study, we can understand that the international cooperation regarding mitigation
and adaptation undertaken by the Japanese government did not conflict with the principles and commitments
established in the international agreement in the international negotiation process and commitment implementation
process.
,,
Therefore, this paper evaluated that Japan s international cooperation did not bring about developed countries
responsibilities as an obstructing factor in global multilateral cooperation towards solving the CC problem, and it,
didn t have adverse effects on the negotiation of the future commitment of developing countries .
On the other hand, this paper points out the policy subject that it is necessary to consider how the Japanese
government situates the CDM and environmental ODA projects that have the direct effects of mitigation andadaptation as their secondary objectives in the system of climate policy.
Moreover, it is requested of the Japanese government that they ensure the implementation of the commitment
regarding mitigation above all. Besides, this paper pointed out that it is indispensable for the Japanese government tocope with the policy issues concerning supplementarity and financial additionality by taking equity into consideration
while restricting efficiency in the commitment implementation process of international cooperation as well as theinternational negotiation process.
3.2 Effectiveness of the evaluation framework
Subsection 3．1 evaluated the international cooperation on the mitigation of and the adaptation to CC, and pointed
out the policy issues resulting from the projects and programs of CDM and environmental ODA undertaken by theJapanese government, which was the case study for applying the evaluation procedure to international cooperation.
This case study focuses on one project for CDM and environmental ODA respectively, which were undertakenby the Japanese government, which is studied as a research subject country. On the premise of such restrictions, this

22

Kiyotaka Nakashima

subsection discusses the effectiveness of the framework based on the analytical results of the case study.
The framework for international cooperation in this paper was established based on the methodological strong
points of EP, namely interdisciplinarity and synthesis. This paper systematically associated multiple elements thatcomprise the framework for international cooperation through the interdisciplinary review of prior studies in thesocial science fields that are concerned with EP as well as the discussion of international negotiation and the content
of the international agreements on CC.
We can evaluate the international cooperation on mitigation and adaptation synthetically at the levels of project,program, and policy through applying an established evaluation framework to the case study.
From the applied case study, this paper was able to evaluate the international cooperation regarding mitigation
and adaptation in which Japan was involved, and to point out the policy subject through the evaluation of the projects
and programs of CDM and environmental ODA.
This result empirically shows the procedure and method to evaluate the international cooperation to combat CC
undertaken by the Japanese government through evaluating the projects and programs of CDM and environmental
ODA.
Therefore, this paper was able to show the effectiveness of the framework despite the limited condition of
focusing on particular projects.
3.3 Research subject of the evaluation framework
Subsection 3．2 pointed out that we were able to evaluate the international cooperation regarding the mitigation ofand adaptation to CC at the levels of project, program, and policy synthetically. It showed the effectiveness of theestablished evaluation framework in this paper under the limited conditions.
On the other hand, there remains the challenge to further the established framework through increasing thenumber of case studies. This aims to improve the framework by expanding the applicable scope of the framework,and thus allowing for more versatility and a wider usage.
This subsection raised five research subjects to further the framework, and discusses its effectiveness andversatility.
The first research subject is to evaluate all projects of CDM and environmental ODA undertaken by Japan as a
research subject country without changing the conditions of the established framework.
We can improve the effectiveness of the framework through evaluating more CDM and environmental ODA
projects undertaken by Japan.
Also, increasing the number of case studies leads to the synthetic evaluation of international cooperation
regarding mitigation and adaptation undertaken by Japan as the research subject country.
We can evaluate other different CDM and environmental ODA projects on the basis of the establishedframework without changing its components, including the issues in international cooperation.
The second is to evaluate the projects and programs of CDM and environmental ODA undertaken by otherdeveloped countries without changing the conditions of the framework.
Similarly, it is also necessary to compare international cooperation of multiple developed countries. By doing
so, we can inspect the framework for the international cooperation through perspectives other than from theevaluation of Japan as the sole research subject country.
These two research subjects increase the number of case studies to which the framework has been applied
without changing the components and conditions of the established framework in this paper.
On the other hand, the following subjects change the framework itself.
The third is to establish issues aside from supplementarity and financial additionality in this paper, and toEvaluation Method of International Cooperation on Climate Change

23

inspect the situation of ensuring these points in the project and program evaluation.
Supplementarity and financial additionality are not the only issues discussed at international negotiations inrelation to the content of the international agreements and the evaluation standards of international cooperation on CC
shown in Figure 1. The framework can be used in a wider context through evaluating international cooperation onother issues.
However, when we change the issues of international cooperation, we cannot apply the framework to a casestudy without the issues relating to the four components shown in Figure 1.
It is necessary to re-examine the discussion of international negotiation and the content of the international
agreements, and review prior studies in the social science field concerning EP when associating the new issues withthe components.
The fourth is to adopt sustainability in addition to the other evaluation standards for international
cooperation, namely equity, efficiency, and effectiveness in this paper.
CC and sustainable development (SD) have been discussed together at international conferences.
For example, UNCED (United Nations Conference on Environment and Development) was held in 1992 after
adopting UNFCCC. This conference emphasized the concept of SD in the Rio declaration on Environment andDevelopment and Agenda 21. The preamble of UNFCCC recalls the provisions of the General Assembly resolution
on UNCED 12.
Also, the effectuation of KP was encouraged at the World Summit on Sustainable Development in 2002. The
Delhi Ministerial Declaration also aimed at coexisting with CC and SD was adopted at COP8 (Eighth session of theConference Of the Parties).
The relationship between sustainability and equity/efficiency is also considered in prior studies in the social
science field concerned with EP on the basis of the present situation13.
It is the academic or practical request that we discuss CC in association with SD. The request is granted throughfurthering the framework for international cooperation by adding sustainability to the established evaluation standards
in this paper 14.
In particular, Article 12.2 of KP is established to achieve SD in developing countries as one of the goals for
CDM projects.
CDM projects that involve developed countries are required to evaluate the achievement of SD in developing
countries in addition to the mitigation effects against CC that come from reducing the emission of GHGs including
CO2.
This paper was not able to focus on sustainability in addition to the other evaluation standards, equity,
efficiency, and effectiveness, due to the need to examine the relatedness of and among the standards.
However, we need to evaluate the international cooperation undertaken by research subject countries with theadditional evaluation standard, sustainability, considering the academic and practical situation and request. The
historical review of the relationship between environment and development will lead to further discussions of therelatedness between CC and SD.
Lastly, the framework for international cooperation in this paper is established for the purpose of evaluating thecommitment implementation and the international cooperation of developed countries on the basis of the discussions
in international negotiations and the content of the international agreements including UNFCCC and KP. Therefore,
we need to change the framework in accordance with the situation of the international system and global multilateral
cooperation on CC after the first commitment period.
For example, it is possible that the mitigation commitment may be imposed on countries with high GHG
emissions (China, India, etc.) and OECD (Organization for Economic Cooperation and Development) member

24

Kiyotaka Nakashima

countries (South Korea, Mexico, etc.) in addition to the Annex II parties of KP after 2012. In this case, we need toconsider changing the framework and its components to evaluate the commitment implementation by developing
countries.
This subsection pointed out five research subjects on the framework for international cooperation on CC
applied in this paper from the perspective of increasing the number of applied case studies and changing components.
These research subjects do not reduce the effectiveness of the framework described in this paper. In fact, theexpanded scope of application implies its wide-ranging applicability to the evaluation of international cooperation on
CC. Such wide-ranging use can improve the effectiveness of the framework.
,
The framework established and applied in this paper is the base to examine the developed countries
commitment to mitigation or adaptation, and to evaluate the international cooperation of developed countries.
Like interdisciplinarity and synthesis, the framework has a qualitative and synthetic meaning based on therelationship of multiple elements that comprise the framework, along with multiple social science fields concernedwith EP.
On this qualitative basis, we can, for example, apply quantitative and specific evaluation methods, such as costbenefit analysis and cost-effectiveness analysis, when using efficiency as the evaluation standard.
Based on such characteristics, we need to further evolve the framework through examining the situation ofinternational negotiations and cooperation, reviewing prior studies in social science fields concerned with EP, andconsidering the research subjects pointed out in this subsection.
This evolution improves the effectiveness of the framework and expands application, and consequently
advances the methodology of evaluation research on climate policy. Also, it can contribute to the evaluation ofcommitment implementation, international cooperation, and global multilateral cooperation on CC.

4. Conclusion
This paper established an evaluation framework based on a social science perspective of EP, and considered theeffectiveness and research subjects of the framework through the applied case studies in order to evaluate theinternational cooperation regarding mitigation of and adaptation to CC.
Based on the result of the case studies, this paper demonstrated the effectiveness of the analytical framework toevaluate the international cooperation regarding mitigation and adaptation, despite the limited condition using one
project each from CDM and environmental ODA.
On the other hand, this paper pointed out the research subject of increasing the number of applied case studies
and changing the components of the framework in order to expand the scope. These additional research subjects do
not reduce the effectiveness of the framework, but rather improves it through expanding its versatility.
The year 2008 is the turning point for global multilateral cooperation towards solving the CC problem, as it is
the starting year of the first commitment period. Every turning point requires an evaluation of multilateral
cooperation thus far observed. This is the academic and practical subject of the EP evaluation theory on CC.
This paper established a framework based on multiple fields of social science and centered on thecharacteristics of the CC problem, using the two strong points of EP studies, interdisciplinarity and synthesis, as a,foundation. By using this framework in case studies, Japan s international cooperation regarding mitigation andadaptation was evaluated. In conclusion, based on the results, this paper showed one of the evaluation methods thatcan respond to the academic or practical subjects.
We need to increase the number of applied case studies and improve the effectiveness and versatility of theEvaluation Method of International Cooperation on Climate Change

25

framework by regarding it as a qualitative and synthetic basis while adopting the quantitative and specific evaluation
methods of social science.
Similarly, we need to change the framework and its components according to the situation of international
negotiations and the changes in institutions of international cooperation in order to solve problems of the academic
and practical subjects on the EP evaluation theory of CC.

Acknowledgements
The author extends his gratitude to the reviewers of this paper who offered their valuable comments.

Notes
1 This paper is a revision of the previous publication, Nakashima (2007a, 2007b).
2 Refer to Rossi, Lipsey, Freeman (2005, p.29) about social intervention.
3 Nishikawa (2005, p.47) points out thatEnvironmental Studies adopts the synthetic and interdisciplinary approach as theresearch method.
4 See Matsuoka and Morita (1998, p.38, p.44).
5 See Rossi, Lipsey, Freeman (2005, p.361).
6 See Miyoshi (2008, p.5), Ryu and Sasaki (2004, p.8), and Ueno (2001, p.18).
7 As the research from politics in EP on CC, for example, see Kanie (2001), Fisher (2004), and Sprinz and Wei(2001). See

Otsuka ed (2004) as an example of a research from law.
8 See Sands (2003), Nishimura (1999, 2000), and Cullet (1999) as examples of research from law. See Ghersi, Hourcade andCriqui (2003), Rose (1990), and Amano (2003) as examples of research from economics.
9 Ministry of Foreign Affairs of Japan and Environmental Agency of Japan (1993, pp.217-218), Chikyukankyohokenkyukai(2003, p.59).
10 See Chapter 2 in Nakashima (2007a) about the consideration of international negotiation process.
11 See Chapter 4 and 5 in Nakashima (2007a) if you would like to know about the case study applied to the evaluation framework
in detail.
12 Chikyukankyohokenkyukai (2003, p.375).
13 See Sungin (2006, p.77), Banuri and Spanger-Siegfried (2002, p.124).
14 Nakashima (2008) considers the relatedness between CC and SD systematically by surveying the researches, for example,
Munasinghe and Swart (2005).

References
Amano, Akihiro. (2003). Kankyo keizai kenkyu

Kankyo to keizai no togo ni mukete. Tokyo, Japan: Yuhikaku.

Banuri, Taril and Spanger-Siegfried, Erika (2002). Equity and the Clean Development Mechanism: equity, additionality,
supplementarity. In Pinguelli-Rosa, Luiz and Munasinghe, Mohan ed. Ethics, Equity and International Negotiations on Climate

Change. Cheltenham, UK: Edward Elgar Publishing Limited．
Chikyukankyohokenkyukai ed. (2003). Chikyukankyojoyakushu

Dai 4 han, Tokyo, Japan: Chuohoki Publishing Co. Ltd.

Cullet, P. (1999). Equity and flexibility mechanisms in the climate change regime：conceptual and practical issues. Review ofEuropean Community and International Environmental Law−Environmental Law in Eastern Europe, 8 (2), 168-179．

26

Kiyotaka Nakashima

Fisher, D.R. (2004). National Governance and the Global Climate Change Regime. Oxford, UK: Rowman & Littlefield
Publishers Inc．
Ghersi, F., Hourcade, J.C., and Criqui, P. (2003).Viable responses to the equity-responsibility dilemma: a consequentialist view.

Climate Policy, 3 (1), 115-133．
Kanie, Norichika. (2001). Chikyukankyogaiko to kokunaiseisaku

Kyotogiteisyo o meguru oranda no gaiko to seisaku. Tokyo,

Japan: Keio University Press Inc.
Matsuoka, Yuzuru. and Morita, Tuneyuki. (1999). Chikyuondankamondai to AIM, Society for Environmental Economics andPolicy Studies ed. Chikyuondanka heno chosen, Toyo Keizai Inc, 38-52.
Ministry of Foreign Affairs of Japan, United Nations Bureau, Economy Division, Global Environment Division andEnvironmental Agency of Japan, Global Environment Department, Policy Planing Division ed. (1993).

Kokurenkankyokaihatsukaigi shiryoshu, Tokyo, Japan: Ministry of Finance Japan, Printing Bureau.
Miyoshi, Koichi. (2008). Hyoka toha nanika . In Miyoshi, Koichi.ed. Hyokaron o manabu hito no tameni. Kyoto,
Japan: Sekaishisosya, 4-22.
Munasinghe, M. and Swart, R.(2005). Primer on Climate Change and Sustainable Development. Cambridge, UK: Cambridge
University Press．
Nakashima, Kiyotaka. (2007a). Evaluation and obstruction factors of the international cooperation for mitigation and adaptation,
on climate change−Environmental policies on Clean Development Mechanism (CDM) and environmental ODA by Japan sgovernment−[in Japanese], Hiroshima City University Ph.D Dissertation.
Nakashima, Kiyotaka. (2007b). Evaluation of international cooperation on mitigation and adaptation of climate change in Japan[in Japanese], Proceedings 4th Spring Conference of the Japan Evaluation Society, 159-166.
Nakashima, Kiyotaka. (2008). Sustainable Development Research on Climate Change [in Japanese], Proceedings of the 12th

Annual Conference Japan Association for Environmental Law and Policy, 51-57.
Nishikawa, Shoko. (2005). Analysis of characteristic of Kankyogaku in Japan-Reviewing books and papers of synthetic andone system science, study and research from view point of environment and environmental problems on Kankyogaku − [inJapanese], Environmental science, 1 (1), 41-51.
Nishimura, Tomoaki. (1999). Climate Change and the System of Global Environmental Convention : An Analysis of Kyoto
Protocol [in Japanese], Mie University the journal of law and economics, 16 (2), 71-95.
Nishimura, Tomoaki. (2000). A Study on the Construction of the Climate Change Convention System：the Solution of the
Problems left in the Kyoto Protocol [in Japanese], Mie University the journal of law and economics, 18 (1), 33-63.
Otsuka, Tadashi. ed. (2004). Chikyuondanka o meguru hoseisaku. Kyoto, Japan: Showado.
Rose, A. (1990). Reducing conflict in global warming policy−The potential of equity as a unifying principle. Energy Policy, 18(10), 927-935．
Rossi, P. H., Lipsey, M. W., Freeman, H. E. (2004). Evaluation: A Systematic Approach Seven Edition. Thousand Oaks,
California: Sage Publications (translated into Japanese by Oshima et.al).
Ryu, Yoshiaki. and Sasaki, Ryo. (2004). Evaluation Theories and Techniques [in Japanese]. Tokyo, Japan: Tagashupan.
Sands, P.(2003). Principles of International Environmental Law Second Edition. Cambridge, UK: Cambridge University Press．
Sprinz, D.F. and Wei , M. (2001). Domestic politics and global climate policy. In Luterbacher, U. and Sprinz, D.F. (Eds),

International Relations and Global Climate Change. Cambridge, Massachusetts: The MIT Press, 67-94．
Sungin, Na (2006). Climate Change and International Cooperation：Efficiency, Equity, Sustainability [in Japanese]. Tokyo,
Japan:Yuhikaku.
Ueno, Hiroshi. (2001). Purojekutohyoka no riron oyobi kongo no kadai, Journal of international development studies [inJapanese], 10(2), 17-48.
(Accepted 13th July, 2010)

27

Publication Policy of the Japanese Journal of Evaluation Studies
Last revised on 15th February 2005
The Purpose and the Name
1．The Japan Evaluation Society (hereinafter referred to as “evaluation society”) publishes “The Japan
Journal of Evaluation Studies (hereinafter referred to as “evaluation study”) in order to widely releaseevaluation studies and outputs of practical activities to domestic and international academic societies,
interested individual and institutions, and contribute to the advancement and prevalence of evaluation
practice.
Editorial Board
2．The editorial board administrates editing of evaluation study based on the editorial policy stated below.
3．The editorial board is formed with less than 20 members of the evaluation society who are assigned by
the board of directors. Terms of editors are two years but can be extended.
4．The editorial board assigns one editor-in-chief, two vice-editors-in-chief, and a certain number ofstanding editors among the members.
5．The editorial board may hold at least one meeting to discuss the editing policy, plans of editorial board,and others.
6．The editorial board reports activities to the board of directors as needed and receives approval. Also it isrequired to report the progress of the past year and an activity plan for the following year at the annual
conference.
7．The editor-in-chief, the vice-editors-in-chief and the standing editors organize the standing committee andadministrate editing on a regular basis.
Editorial Policy
8．The evaluation study, as a principle, is published twice a year.
9．The evaluation study is printed on B5 paper, and either in Japanese or English.
10．Papers published in the evaluation study are categorized as five types;
10.1．Review
10.2．Article
10.3．Research note
10.4．Report
10.5．Others
11．The qualified contributors are members of the evaluation society (hereinafter referred to as “members”)
and persons whose contribution is requested by the standing editors. Joint submission of members andjoint submission of non-members with a member as the first author are accepted. Submission by theeditors is accepted.
12．Submitted manuscripts are treated as the above categories, however, the standing editors will decide
based on the application of the contributors and the following guidelines;
12.1．“Review” is a paper, which provides an overview of evaluation theory or practice. The editorial board
will make the decision regarding publication.
12.2．“Article” is considered as a significant academic contribution to the theoretical development ofevaluation or understanding of evaluation practice. The standing editors committee makes adoption
judgments following the referee-reading process described in the next section.
12.3．“Research note” is a discussion equivalent to the intermediate outputs of a theoretical or empiricalenquiry. The standing editors committee makes adoption judgments following the referee-reading
process described in the next section.

28

12.4．“Report” is the study report related to a practical evaluation project or evaluation. The standing editorscommittee makes adoption judgments following the referee-reading process described in the next
section.
12.5．“Others” includes requested papers for special editions organized by the editorial board andannouncements from editorial board to members regarding publication.
13．The editorial board selects two referee readers. For the “article”, the editorial board makes adoption
judgments referring to the results from referee readings and comments provided by one editor assigned by
the editorial board. For “review”, “research note”, “report” and “others”, the editorial board makes
adoption judgments referring to the results from referee readings.
14．When editors submit a manuscript, the editors are not allowed to attend any of the standing editorscommittee meetings or editorial board meetings regarding the manuscript.
15．The standing editors have alternative of approval or not-approval for adoption judgment of manuscripts
submitted to any categories. However exception is permitted if the editorial board approves the
publication after minor rewrite. Even if the manuscripts are considered insufficient as an “article”,
standing editors can decide whether the manuscripts are published as a “research note” or “report” if theauthors wish to publish.
Formulation and Release of Submission Procedure
16．The editorial board formulates the submission procedure based on the editorial policy described above
and release after approval from the board of directors.
Distribution
17．The evaluation study is distributed to all members for free and distributed to non-members for a charge.
Distribution of the Printed Manuscript
18．30 copies of the respective paper are reprinted and distributed to the authors. The authors must cover any
costs incurred by author,s requests for printing more than 30 copies.
Release on the Internet
19．The papers published in the evaluation study are released on the internet with approval from the authors.
Copyright
20．Copyright of papers which appear in the evaluation study is attributed to the respective authors. Editorial
right is attributed to the evaluation society.
Office21．The office is in charge of administrative works for editing, distribution, and accounting.

29

Information for Contributors(For English Papers)
Last revised on 29th September 2008
1．“The Japanese Journal of Evaluation Studies” is the publication for reviews, articles, research notes, andreports relating to evaluation.
2．“The Japanese Journal of Evaluation Studies” is primary published to provide opportunities for membersof the Japan Evaluation Society (hereinafter referred to as “members”) to exchange findings, and tocontribute to further development of the study of evaluation both domestically and internationally. As aprinciple, this journal publishes the contributions submitted by the members. With the exception ofrequested papers, the first author must be a member. A submission (as the first author) is limited to one
manuscript that has not been published or submitted in any form for another journal of academic
association etcetera.
3．Adoption judgments of the manuscript are made at the discretion of the editorial board. Comments fromtwo referee readers who are appointed for every manuscript are referred to in the screening process (theeditorial board requests referee readers without notifying the author of manuscript).
4．Payment for the manuscript is not provided.
5．Papers published in “The Japanese Journal of Evaluation Studies” are released on the Internet at
homepage of this academic society.
6．Regarding submission, manuscripts must be identified as one of the following categories: 1) article, 2)
review, 3) research note, 4) report, and 5) others. However, the final decision of the category is made by
the editorial board.
“Article” is considered as a significant academic contribution to the theoretical development of evaluationor understanding of evaluation practice.
“Review” is a paper which provides an overview of evaluation theory or practice.
“Research note” is a discussion equivalent to the intermediate outputs of a theoretical or empirical study
in the process of producing an “article”.
“Report” is the study report related to a practical evaluation project or evaluation.
“Others” are manuscripts for special editions requested by the editing committee.
7．Manuscript Submission(1) Manuscripts may be written in either Japanese or English.
(2) Correction by the author is only for the first correction.
(3) English manuscripts should be submitted only after the English has been checked by a native speaker.
(4) Submit four hard copies (A4 size) of the manuscript. Contact information including mailing address,
telephone number, fax number, and e-mail address, and the category of the manuscript should beclearly stated.
For approved manuscripts, after necessary rewriting, the author needs to submit two hard copies of the
final paper as well as a text file saved on a DOS/V formatted floppy disk. Original figures, charts, andmaps should be provided.
(5) Total printed pages should not exceed 14 pages. Any cost incurred by printing more than 14 pages
must be covered by the author.
(6) The layout for English papers should be 30 mm of margin at left and right side, 10pt for font size, 43

30

lines on A4 paper (about 500 words per page). An abstract of 150 words should be attached to the front.
14 pages are equivalent to 7,000 words but the body should not exceed 6,000 words to allow for the title,
header, figure, chart, footnotes, and references. Please note that the number of pages may be more than
expected depending on the number of figures included.
8．Mailing address
Office of Japan Evaluation Society at International Development Center of Japan
Hitachi Soft Tower B 22nd Floor, 4-12-6, Higashi-Shinagawa, Shinagawa, Tokyo,
140-0002, Japan
Phone: +81-3-6718-5931, Facsimile: +81-3-6718-1651
E-mail: jes.info@evaluationjp.org

31

Writing Manual of the Japanese Journal of Evaluation Studies(For English Papers)
Revised on 18th September 2002
1．Text, Charts, Figures, Graphs, Diagrams, Notes, and References(1) The paper should be written in the follow order:First page: Title; the author,s name; Affiliation; E-mail address; Abstract (150 words); Keywords (5 words)
Second page: The main text; acknowledgement; notes; references(2) Section of the text should be as follow:
1.
1.1
1.1.1
1.1.2
(3) Source of the charts, figures, graphs, and diagrams should be clarified. Submitted charts and others will be
photoengraved, therefore it is important that the original chart is clear. Pictures shall be treated as figures.
Figure 1 Number of Students in the State of ○○Note:
Source:Table 1 Number of Accidents in the State of ○○Note:
Source:(4) Citation of literature in the text should be, (Abe 1995, p.36) or (Abe 1995).
(5) Note in the text should be, (------.1 )32(6) Note and references should be written all together in the end.
Note
1 --------.
2 --------.
(7) Reference should list the literature in alphabet order, and arranged in chronological order. Follow theexamples:Book: author (year of publication). Title of the book. Published location: publishing house.
(e.g.) Rossi, P. H. (1999). Evaluation: A Systematic Approach 6th edition. Beverly Hills, Calif: Sage
Publication.
Article from magazine: author (year of publication). Title. Title of the magazine, volume (number), pagepage.
(e.g.) Rossi, P. H. (1999). Measuring social judgments. American Journal of Evaluation, 15(2), 35-37.
Article in Book: author (year of publication). Title. In editor (Eds.), Title of the book. Published location:
publishing house, page-page.
(e.g.) DeMaio, T. J., and Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in thefield. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitiveand communicative processes in survey research. San Fransisco, Calif: Jossey-Bass, 177-196.
Book by two authors: surname, first name, and surname, first name. (year of publication). Title of the book.
Published location: publishing house.
(e.g.) Peters, T., and Waterman, R. (1982). In Search of Excellence: Lessons from America,s Best Run
Companies. New York: Harper & Row.
Book by more than three authors: surname, first name, surname, first name, and surname, first name. (year of
publication). Title of the book. Published location: publishing house.
(e.g.) Morley, E., Bryant, S. P., and Hatry, H. P. (2000). Comparative Performance Measurement.
Washignton: Urban Institute.
(note 1) If some references are from the same author with the same publication year, differentiate by adding
a,b,c as (1999a), (1999b).
(note 2) If the reference is more than a single line, each line from the second should be indented by threespaces.
(e.g.) DeMaio, T. J., and Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in thefield. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitiveand communicative processes in survey research. San Fransisco, Calif: Jossey-Bass, 177-196.

33

Referee-Reading Guideline
The Japanese Journal of Evaluation Studies Editorial Board,
The Japan Evaluation Society
Approved on 10th September 2005
1．Content of the Referee-Reading Guideline
This Referee-Reading Guideline is to provide explanation of the main publication judgment, procedure of the refereereading, to the members who submit the manuscript and for the members who are requested to conduct refereereading in order to carry out the procedure efficiently and effectively.

2．Purpose of Referee-Reading and the Responsibility of the Author
Referee-reading is necessary for the editorial board to make decisions of whether submitted manuscripts areappropriate to publish in the Japanese Journal of Evaluation Studies or not.
If there is doubt or obscurity identified in manuscripts during the referee-reading corrections may be required.
Therefore, referee-reading also contributes to the improvement of the submitted manuscripts. However, although themanuscripts are requested corrections, the author is still solely responsible in regards to the contents and it is not
attributed to the referee-readers.
Referee-readers are two persons who are requested by the editorial board depending on the specialty or the field of
the submitted manuscript. People who are not members of this academic society also may be requested.

3．Items of Consideration in Referee-Reading
Five points are considered in referee-reading, however, the importance of each may be different depending on thetype of manuscript.
(1) Importance and utility of the theme(2) Originality of the study(3) Structure of the logic(4) Validity of verification and methodology(5) Contribution to evaluation theory and practice-

For the article, all of above five are considered.
For the research note, especially (1), (2), (3), and (4) are considered.
For the report, especially (1), (3), and (5) are considered.
For the review, especially (3) and (5) are considered.

34

4．Attentions in submission of manuscript
Besides above five viewpoints, basic completeness as a paper is also considered, for example;- appearance of the paper is organized- written according to the writing manual- described simply and distinctive- verification data is appropriately used
- notes and references are corresponding with the text
- terminology is appropriately used
- no wording and grammatical mistakes- no errors and omission- no punctuation mistakes- expression in English abstract is appropriate- word count is according to the manual
The above mentioned forms and contents are also considered. There have been cases in which graduate students andpractitioners posted without organizing the manuscripts as a paper. On those occasions, referee-reading was notconducted. Necessary consultation is strongly recommended prior to submission.

5．Judgment Cases in Referee-Reading(1) In the case of the manuscript which is considered acceptable for the publication but is not yet complete:
The referee reader should evaluate carefully whether the paper can contribute to the development ofevaluation theory or evaluation studies.
-

Verification is lacking but the theory and formulation are useful for academic development.
Analysis lacking but useful for formation and promotion of new theory.
The literature review is not of a high standard but, the overall study is meaningful.
Comparative study is not up to standard but is meaningful as an example of application.
Analysis is lacking but it is meaningful as an evaluation of socially and historically important cases.
Analysis is lacking but it is meaningful as an evaluation of particular social activities.
Organization and expression are not up to standard as a paper but the contents are worthy to evaluate.
Logic is not strong enough but useful in practice.
The paper has significance as a report.

(2) In case of the manuscript which is considered as difficult for publication:- Awareness of the issue or setting of the problem is indecisive.
- Understanding or analytical framework of notion of basic terminology is indecisive or inappropriate.
- There is a lack in credibility of data for the grounds of an argument.
- There is no clear point of an argument or appropriateness of proof.
- Organization of the paper and presentation (terminology, citation, chart, etc) are inappropriate (or notconsistent).

35

6．Judgment
The final decision will be made on publication at the standing editors committee following one of four patterns (listed
below). However, these judgments are not based on the number of errors but on the strength of the overall report. In
the case of (3) and (4), there is a possibility to be published as a different type of paper. If it is published as a different
type of paper, major rewrite concerning the number of words may be required.
(1) The paper will be published.
(2) The paper will be published with minor rewrite.
(3) The paper will be published with major rewrite, however as a different type of paper (review, article,
research note, or report).
(4) The paper will not be published; however there is the possibility that it will be published as a different typeof paper (review, article, research note, or report).

Japanese Journal of Evaluation Studies, Vol. 10, No. 2, September, 2010

Published by

The Japan Evaluation Society
c/o International Development Center of Japan
Hitachi Soft Tower B 22nd Floor, 4-12-6
Higashi-Shinagawa, Shinagawa
Tokyo 140-0002, Japan
Phone: +81-3-6718-5931, Facsimile: +81-3-6718-1651
E-mail: jes.info@evaluationjp.org

Printed by

Kenkosha, Co., Ltd.

Japan Evaluation Society. Reproduction of material contained in this document
may be made only with written permission of Japan Evaluation Society.

ISSN 1346-6151

Japanese Journal of Evaluation Studies

Japan Evaluation Society