Japanese Journal of Evaluation Studies
Vol. 12, No. 2, November 2012English Version

CONTENTS
Mini Special Issue: Evaluation Education in Graduate Programs
Ryo Sasaki
Graduate Education 1: Evaluation Education in the United States:
Can Evaluation Be a Discipline?……………………………………………………………………1
Keiko Kuji-Shikatani, James McDavid, J. Bradley Cousins, Heather Buchanan
Consortium of Universities for Evaluation Education and its Impacts on
Professionalizing Evaluation in Canada …………………………………………………………11
Research Note
Satoshi Morita
Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance …………………………………………23

Publication Policy of the Japanese Journal of Evaluation Studies ………………………………………41
Information for Contributors (For English Papers) ………………………………………………………43
Writing Manual of the Japanese Journal of Evaluation Studies(For English Papers) ……………………45
Referee-Reading Guideline ………………………………………………………………………………47

1【Research Note】【】

Graduate Education 1: Evaluation Education in the United States:
Can Evaluation Be a Discipline?
Ryo Sasaki
International Development Center of Japan Incorporated (IDCJ)
sasakiryo@gmail.com

Abstract
The purposes of this study are to understand how evaluation education program is different from other programs andto make clear the general structure of evaluation education programs which might be generalized globally. I
examined and categorized the courses offered by 48 evaluation education programs in the U.S. that LaVelle andDonaldson have identified (2010). Then one doctoral program and one public policy program are reviewed in order tomake clear new aspects of evaluation education program as a start-up discipline.

Keywords
Evaluation education, Evaluation-specific logic, models, methods, public policy analysis

1. Introduction
Evaluation education at graduate school is one of the hottest topics in the field of evaluation. It is frequently observedthat evaluation had emerged as one solid academic discipline in the last few decades. The purposes of this study are
to understand how evaluation education program is different from other programs and to make clear the general
structure of evaluation education programs which might be generalized globally. For these purposes, I examined thescene of evaluation education at graduate schools in the U.S.. This paper consists of four parts. First is the overview
of graduate programs in the U.S. based on the past studies. Second part is the overview and examination of one of themost solid evaluation education programs, Interdisciplinary PhD in Evaluation at Western Michigan University.
Third is the comparative analysis of this new discipline (evaluation) and a traditional discipline (public policy
analysis) in order to make clear the difference between them. The last part is the discussion about the future of thisnew discipline and the future prospect of evaluation education.
A few notes should be mentioned. This study is based on the past contribution of other researchers as many,
researchers do. Also, this study is heavily based on the author s personal experience including study experience in the,
doctoral program at Western Michigan University and the master s program at New York University. It would be safe
to say that this paper is just a start and offers a basis for discussing how we make progress in our endeavor forJapanese Journal of Evaluation Studies, Vol.12, No.2, 2012, pp.1-10

2

Ryo Sasaki

evaluation education in the future.

2. Current trend of evaluation education in the U.S.
LaVelle and Donaldson (2010) recently made one excellent review of the evaluation education programs at U.S.
graduate schools. They identified 48 graduate programs by using very systematic internet search. Some of theirconclusions are as follows (LaVelle & Donaldson 2010, p.15).
● 9 programs offer a master,s specialization only (18.8%), 14 offer a doctoral specialization only (29.2%),,
and 25 offer specialization at both the master s and doctoral levels (52.1%)1. (See Figure 1)● More than half of the programs are taught in schools of education (n=29, 60.4%), while others are taught
in schools of educational psychology (n=7, 14.5%), psychology (n=5, 10.5%), public policy (n=4, 8.3%),
criminal justice (n=1, 2%), applied sociology (n=1, 2%), and interdisciplinary study (n=1, 2%). (SeeFigure 2).
Figure 1

Level of evaluation education at U.S. graduate schools(n=48)a

30
25
25
20
14

15
10

9

5
0

,
Master s specialization

Doctoral specialization,
Both the master s anddoctoral levels

Source: Data are adapted from LaVelle and Donaldson (2010)

Figure 2

Schools hosting evaluation education at U.S. graduate schools

Criminal justice,
1

Appliedsociology, 1

Interdisciplivary
study, 1

Public policy, 4Psychology, 5

Educational
psychology, 7

Source: Data are adapted from LaVelle and Donaldson (2010)

Education, 29(n=48)

Graduate Education 1: Evaluation Education in the United States:
Can Evaluation Be a Discipline?

●3

3 programs are regarded as large programs offering more than 7 evaluation-specific courses; 14 are
medium programs offering 4 to 6 evaluation-specific courses; and 30 are small programs offering 2
or 3 evaluation-specific courses2. The study does not count programs offering only one evaluation-specificcourse, which numerous and various schools and departments do, such as education, health, social work,
public administration, and international development.

Since the number of graduate programs has increased compared to the past similar studies3, LaVelle andDonaldson make a conclusion at the end of their paper (p.21) that By all measures, evaluation is a growing
profession and many universities are now involved in the preservice training of evaluations .
In addition, I tried further analysis based on the list of evaluation-specific courses that LaVelle and Donaldson
have identified. Based on this, I set general categories of courses4 as I. Foundation; II. Models and approaches; III.
Methods; IV. Applications (subject-matter); V. Applications (field experience); VI. Advanced seminars; and VII.
Others. Sub-categories are set under those seven categories. The numbers of courses for each category and associatedsub-category are presented in Table 1.

Table 1Categories of courses offered by evaluation education

I. Foundation● Introduction of (program) evaluation● Foundations of (program) evaluation
II. Models and approaches● Evaluation models● Evaluation theories● Evaluation methods (general)● Program evaluation● Program evaluation (advanced)
III. Methods● Evaluation designs / procedures● Measurement and evaluation● Qualitative evaluation / methods● Quantitative evaluation / methods● Cost-effectiveness● Needs assessment
IV. Applications (subject-matter)● Evaluation of social programs● Evaluation of health programs● Evaluation of educational programs● Personnel evaluation
V. Applications (field experience)● Practicum / Capstone● Internship
VI. Advanced seminars● Seminars / Workshops
VII. Others● Others / Unable to be categorized

(9)
5
4
(45)
6
9
6
17
7
(23)
5
8
2
1
5
2
(25)
3
1
16
5

Total number
Source: Data are adapted from LaVelle and Donaldson (2010)

18
6
(19)
19
(20)
20
(165)

4

Ryo Sasaki

Several difficulties were found during this categorization work.
● Foundations of evaluation is so-called the pure part of the discipline. It should offer definitions of thecentral concepts of evaluation (e.g., what is evaluation ?; difference between evaluation and research;and common procedures for evaluation). This is the essential part if a field is to be called as an academic
discipline. However, it is not clear whether such topics are taught in the courses named introduction of(program) evaluation . If these courses teach just an overview of the entire degree program, they do notdeserve to be called as a foundation. It means such programs lack core or pure part of the discipline.
● I categorized courses titled program evaluation as II. Models and Approaches. Program evaluation
courses are offered at many schools and often divided into two levels, Basic and Advanced (or I and II).
Also, it is expected that there are great varieties and diversities in contents offered by different courses
under this single name. Some courses offer foundations (as explained above) and models/approaches if the
instructor uses model-oriented textbooks (e.g., Owen (2006); Fizpatrick, Sanders & Worthen (2004)).
Others purely offer designs and procedures for evaluating programs if the instructor uses methodoriented textbooks (e.g., Bingham & Felbinger (1999) and textbooks influenced by the tradition ofCampbell and Stanley (1966)). It is difficult to distinct them by only names of courses.
● Only one course is counted as quantitative evaluation/methods . However, it is generally recognized thatmost evaluation education programs require students to take at least one quantitative course for thecompletion of the program. It is understood that those courses do not include the word evaluation , andinstead, they are offered under different names such as basic statistics or statistical data analysis .
They are not captured in the procedure taken by LaVelle and Donaldson because they identified only
courses whose titles include the word evaluation .
Although the above concerns exist, the categorization indicates a kind of common structure of evaluation
education programs. A need for further research clearly exists, and it should conduct in-depth analysis of respective
evaluation education programs.

3. One solid exemplar: Interdisciplinary PhD in Evaluation at Western Michigan University.
It is quite difficult to understand the actual contents if we examine only titles and brief description of respective
courses. Fortunately, the author (Sasaki) has completed one of the most solid evaluation education programs, theinterdisciplinary PhD in evaluation organized by the Evaluation Center at Western Michigan University. Thus, I will
explain what the program actually offers and degree of achievement of its goals.
As a start, let us see the stated mission and the officially offered course structure. First of all, the mission of thedoctoral program is defined as to develop evaluation scholars and practitioners who have deep knowledge ofevaluation theory, methodology, and practice as well as superior critical thinking and problem-solving skills . In
short, it states that the program will develop and nurture thought leaders in the field. Secondly, the course
structure is presented in Table 2.
The first and core of the program is Foundations of Evaluation under the cluster of I. Evaluation. This course isdesigned to provide an overview of both past and contemporary perspectives on evaluation theory, method, andpractice. Topics include: fundamental evaluation concepts and definitions; the basic logic of evaluation; distinctions
between evaluation and applied social science research, and evaluation-specific methods.
I observed quite many evaluation scholars, including some well-known scholars, fail to answer the differenceGraduate Education 1: Evaluation Education in the United States:
Can Evaluation Be a Discipline?

Table 2

5

Course structure, Interdisciplinary PhD in Evaluation

Name of Program: Interdisciplinary PhD in Evaluation, Western Michigan University
I.
Evaluation competencies< Interdisciplinary evaluation courses>● Foundations of Evaluation (1 course)● Interdisciplinary Seminar in Evaluation (1-3 courses)● Practical Evaluation Experience (3 courses)< Evaluation-related courses offered by other graduate schools)● Program / Intervention Evaluation (1-2 courses)● Social, Political, and Cultural Context of Evaluation (1-2 courses)● Specialized Evaluation-Related Courses (4-6 courses)
II.
Research Methods
The following courses are the ones the author (Sasaki) selected and completed.
● Research Design (ANOVA)● Small N Research Design● General Linear Model (GLM)● Qualitative Research Methods● (Two transferred courses from my masterﾕs completion)
III.
Cognate Areas
Students choose one area and make specialized expertise. Possible cognate areas
include:● Health● Education● Social work
● International development● Organizational change, etc.
IV.
Dissertation● Doctoral Dissertation
Source: The Evaluation Center,s website ( http://www.wmich.edu/evalphd/ ) and the author,s experience.

between evaluation and research, or the difference between evaluation and policy analysis. Some people explain that
evaluation is applied and research is basic, and some others explain evaluation is retrospective and policy analysis isprospective. Those distinctions are relative nature and fail to make clear the difference between them. Evaluation isdefined as determination of values (i.e., merit, worth and significance) of things. Thus, evaluation consists of two
premises; (i) factual premise and (ii) value premise. In other words, evaluators should identify what the fact is andthen make a value judgment based on the identified fact. Needless to say, several broader definitions have been
proposed, but it is now generally agreed that something like value determination is the core of evaluation. Shadish,Cook, and Leviton published a landmark book of the field (1991) and introduced seven pioneers in the field. This
work of Shadish, Cook, and Leviton followed the historical development of the field. They carefully reviewed theworks of seven pioneers in the field of evaluation. They are as follows.
In this book, they clearly concluded that nearly all the theorists in this book agree that evaluation is about
determining value, merit, or worth, not just about describing programs (p. 49). This was stated more than twenty
years ago from now. Based on this fundamental definition of evaluation, various basic logics of evaluation andevaluation-specific methods are taught in the course of Foundations of Evaluation.
Then students take the second course, Interdisciplinary Seminar in Evaluation, which provides major debates
and the current themes in the field. The third course, Practical Evaluation Experience, asks students to take
opportunities to actually conduct an evaluation. A group of students conducted an evaluation of an international

6

Ryo Sasaki

Table 3

Stages in the evolution of theories of evaluation

Stage one theories: Bringing truth to social problem solving
Scriven, M.:The science of valuing
Campbell, D. T.: Methodologist of the experimenting society
Stage two theories: Generating alternatives emphasizing use and pragmatism
Weiss C.H.: Linking evaluation to policy research
Wholey, J.S.: Evaluation for program improvement
Stake, R.E: Responsive evaluation and qualitative methods
Stage three theories: Trying to integrate the past
Cronbach, L. J.: Functional evaluation design for a world of political accommodation
Rossi, P.H.: Comprehensive, tailored, theory-driven evaluation−A smorgasbord of options
Source: Shadish, Cook & Leviton (1991),
NGO s projects in Africa while I was taking this PhD program.
The PhD program asks students to take evaluation-related courses of other graduate schools of the University.
They include sector-specific evaluation courses, such as evaluation of health programs, education programs, social
work programs, and international development.
The second cluster is Research Methods. Under this cluster, students should take a set of courses in researchmethods. As previously explained, evaluation consists of two works; factual premise (fact identification) and value
premise (value determination). This cluster is expected to contribute to the former, fact identification. This cluster
equips students with solid knowledge and social science research skills. It includes both qualitative and quantitativeknowledge and skills. These courses are offered by related schools of the University, including School of Education.
The author took two advanced statistical courses, Research Design (ANOVA) and General Linear Modeling (GLM),
from the School of Education.
The third cluster is Cognate. Under this cluster, students obtain subject-matter knowledge. The subjects include
education, health, social work, public administration, international development, engineering and others. As taught inFoundations of Evaluation, people become able to conduct evaluation if they are trained in the basic logic ofevaluation, but it is risky to offer recommendations, especially macro-level recommendations such as whether theevaluand (=the evaluated program) should be continued or not. In order to offer plausible and feasible
recommendations, people should have three kinds of knowledge; subject-matter knowledge, solid evaluative
conclusions, and the knowledge about the organization5. This cluster partially satisfies these conditions. Needless tosay, the possibility that an evaluator generates a good evaluative conclusion increases if s/he has subject-matterknowledge.
Just recently, the program started offering several new courses. One of them is Experimental and QuasiExperimental Designs for Applied Research and Evaluation. This course offers systematic training in the principles
of designing experimental, quasi-experimental and, to a lesser extent, non-experimental investigations for appliedresearch and evaluation. Another one is Cost Analysis, which provides an overview of cost analysis in evaluation,
including cost-effectiveness analysis, cost-benefit analysis, and cost-utility analysis. It seems that these courses
offered by the program (not by other schools in the University) fill the requirements for the second cluster ( II.
Research Methods).

Graduate Education 1: Evaluation Education in the United States:
Can Evaluation Be a Discipline?

7

4. Comparison of new and traditional disciplines: Evaluation and Public Policy Analysis
It seems that the education program which is the most similar (or the closest) to evaluation education, outside ofeducation field, is public policy analysis. There are more than 120 schools of public administration in the U.S.; they
offer PhD or Master in public administration (MPA), and many of them offer specialized courses in public policy.
Some offer PhD or Master in public policy (MPP). Annual ranking of such schools with public policy analysis
specialization is published in U.S. News and World Report6.
Fortunately, I have been trained in one of those schools, Robert F. Wagner Graduate School of Public Service,
New York University (NYU), which is one of the largest private universities located in New York City. I believe it isa good exemplar for examining how evaluation is different from this traditional discipline.
The courses that I completed at NYU are as follows.
This public policy program is not included in the evaluation education programs identified by LaVelle andDonaldson (2010) because the program has only one course whose title has the word evaluation among core andrequired specialization courses. Program Analysis and Evaluation (in II. Required specialization courses) is the one
including the word evaluation .
It is true that I cannot directly compare the program at WMU and the program at NYU because they are not at,
the same level of education (PhD program and Master s program). However, it would be worth comparing them for
examining characteristics of the newly emerging discipline, evaluation.
One similarity between the examined evaluation program and this traditional public policy program is that theyboth utilize analytical tools of economics and statistics. They share the same tools and techniques including
regression analysis (statistics) and cost-efficiency analysis (economics). Thus evaluation programs and public policy
Table 4,Course Structure of Public Policy Program (Master s level)

Name of Program: Pubic Policy Analysis specialization, Master of Public Administration(M.P.A.), New York University
I.

II.

III.

IV.

Core courses● Introduction to Public Policy● Statistical Methods● Microeconomics● Managing Public Service Organizations● Financial Management
Required specialization courses● Public Economics and Finance● Program Analysis and Evaluation● Multiple Regression and Introduction to Econometrics● Policy Formation and Policy Analysis
Electives (Education, Health, International Development, NPO Management, etc.)
The following courses are the ones the author (Sasaki) selected and completed.
● Economic Development● International Finance● Understanding World Macroeconomy● Regression and Multivariate Analysis● Evaluation of Health and Social Service Programs (Master thesis)
Capstone● Advanced Project in Public Policy

Source: Robert F. Wagner Graduate School of Public Service, New York University (http://wagner.nyu.edu/academics/)

8

Ryo Sasaki

programs do not seem much different if we compare only analytical tools offered.
On the other hand, the most important difference is that this public policy program seems not to distinguishevaluation from the general concept of policy analysis. As you see in Table 4, evaluation is regarded as just a part ofone specialization course, Program Analysis and Evaluation. The textbook I used in this course was Evaluation: A
Systematic Approach 5th edition (Rossi & Freeman1993). It is broadly recognized that this textbook is under strong,
influence of Campbell and Stanley s tradition, and the book defined evaluation as systematic application of social
research procedures for assessing the conceptualization design, implementation, and utility of social intervention
programs (p.5). This definition does not touch upon the value determination issue, and this definition might also be
regarded as a definition of public policy analysis. The conclusion is that the emphasis of evaluation-specific logic,
which essentially distinguishes evaluation from general public policy analysis, is not enough or weak in this program.
And this conclusion would be applied to public policy programs in general to a certain degree.

5. Conclusion and limitations of the study
In this paper, I examined and categorized the courses offered by 48 evaluation education programs in the U.S. Then
one doctoral program was examined to understand its structure and contents. Finally one public policy program wasbriefly reviewed in order to make clear new aspects of evaluation education programs. As previously stated, thepurpose of this study is to understand how evaluation education program is different from other programs and to
make clear the general structure of evaluation education program which might be applied to other parts of the world
including Japan. I hope this study contributes to this purpose.
Scriven (1991) once explained cores (= a general set of components) of an educational discipline in his
landmark book (p.23). His writing is so sophisticated that it is sometimes difficult to clearly understand. However, in
my understanding, the cores he offered are; (i) metatheory, (ii) general theoretical component of the field, (iii)
general methodology of the field, and (iv) a little assorted application work.
Evaluation education programs would consist of the following courses if we follow the list of cores of adiscipline. It generally matches the list of the categories that I developed in this paper. Needless to say, the difference,in levels (doctoral, master s, etc.) should be kept in mind if someone is to develop an actual evaluation education
program.
● Metatheory:● Theory:Foundations or evaluation-specific logics
Models and approaches, such as the categorization offered by Fitzpatrick, Sanders, andWorthen (2004) and the categorization by Owen (2006)● Methodology: Methods or analytical tools, such as statistics (basic and advance), cost-benefit analysis,
qualitative methods (observation, interview and text analysis), etc.
● Applied
Subject-specific evaluation practice, such as educational evaluation, health evaluation,
evaluation:social work evaluation, international aid evaluation, etc.
As stated in the beginning, this study is just a start of establishing general structure of evaluation education thatis applicable globally. In fact, this study examined only two graduate programs that the author has completed. It is,necessary to study more samples and conduct in-depth analysis for both master s and doctoral programs. Onceenough analyses are made, research on general structure of evaluation education program should be re-examined. It
would be a long road, but once it is completed, evaluation will be not a new discipline any more but a well-Graduate Education 1: Evaluation Education in the United States:
Can Evaluation Be a Discipline?

9

established traditional discipline.

Notes,
1 The original text (LaVelle and Donaldson, 2010) indicated 12 programs offer a master s specialization only (18.7%), 16 offer a,doctoral specialization only (29.2%), and 25 offer specialization at both the master s and doctoral levels (52.1%). If they arecorrect, the total number should be 53 (=12+16+25) which is more than 48. Thus the Author (Sasaki) counted numbers by handagain using the table included in original text (Table 2. in P.16, LaVielle and Donaldson, 2010) and concluded slightly different
numbers and ratios.
2 If we add all three numbers, the total number becomes 47 (=3+14+30). One is missing, but it was impossible to identify it andthe author (Sasaki) decided to maintain the original counts made by LaVelle and Donaldson.
3 One of the past similar studies is Engle, Molly, and James W. Altschuld (2004) An update on university-based evaluation
training, In The Evaluation Exchange, IX-4
4 Scriven (1991, p.23) proposed a general set of components which a discipline consists of, as mentioned in the conclusion partof this paper. The categories in Table 1 generally follow his proposal but are further extended, such as the addition of V. VI,
and VII.
5

Macro-recommendations−which are about the disposition or classification of the whole program (refund, cut, modify,
export, etc.−which we might be call external management recommendations, or dispositional recommendations)−are usuallyanother matter. These are important decisions serviced by and properly dependent on, summative evaluations, but making
recommendations about the evaluand is not intrinsically part at of the task of evaluation.

In Scriven, M. (2011).Key

Evaluation Checklist (KEC) [Edition of July 29, 2011]. http://michaelscriven.info/images/KEC̲7.29.11.2011.pdf
6 See the following website for the full list.
http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-public-affairs-schools/public-policy-analysisrankings
7 Their categorization is: Objective-oriented evaluation approaches; Management-oriented evaluation approaches; Consumeroriented evaluation approaches, Expertise-oriented evaluation approaches; and Participant-oriented approaches.
8 His categorization is: Proactive evaluation; Interactive evaluation; Monitoring evaluation; and Impact evaluation.

References
Bingham & Felbinger (1999). Evaluation in Practice: A Methodological Approach. CQ Press, an imprint of Sage Publications,
Inc. CA: Newbury Park
Campbell, D. T., and Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Skokie, IL: Rand McNally.
Fitzpatrick, J.L., Sanders, J.R., and Worthen, B.R. (2004). Program Evaluation (3rd ed). Boston, MA: Pearson Education, Inc.
LaVelle, J. and Donaldson, S.I. (2010). University-Based Evaluation Training Programs in the United States 1980-2008: An
Empirical Examination In American Journal of Evaluation, 31(1), 9-23.
Owen, J. (2006). Program Evaluation: Forms and Approaches (3rd ed). Allen and Unwin. Australia: Crows Nest NSW.
Rossi, P. H., and Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Thousand Oaks, CA: Sage.
Sasaki, R. (2010). The Basic Logic of Evaluation: The Base for Evaluation Study. Tokyo, Japan: Taga-shuppan (written in
Japanese).
Shadish, W. R., Cook, T. D., and Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Thousand Oaks,
CA: Sage.
Scriven, M. (1991). Evaluation Thesaurus (4th ed). Newbury Park, CA: Sage Publications, Inc.

10

Ryo Sasaki

Scriven, M. (2011). Key Evaluation Checklist (KEC) [Edition of July 29, 2011].

http://michaelscriven.info/images/KEC̲7.29.11.2011.pdf(Accepted 28th May, 2012)

11【Report】

Consortium of Universities for Evaluation Education and its
Impacts on Professionalizing Evaluation in Canada
Keiko Kuji-Shikatani

James McDavid

Ontario Ministry of Education
Keiko.Kuji-Shikatani@ontario.ca

University of Victoria
jmcdavid@uvic.ca

J. Bradley Cousins

Heather Buchanan

University of Ottawa
bcousins@uottawa.ca

Jua, Management Consulting Services
hbuchanan@jua.ca

Abstract
The Consortium of Universities for Evaluation Education is an unique collaborative partnership of post-secondary
institutions, government and the Canadian Evaluation Society in offering expanded graduate-level evaluation courses
and programs for current and prospective evaluation practitioners (CUEE, 2011a) transforming evaluation educationin Canada. CUEE experience is examined to share how a vast and diverse country with limited graduate-levelevaluation education opportunities can offer learning options to promote the practice of ethical, high quality andcompetent evaluation for the betterment of society.

Keywords
Evaluation education, evaluation competencies, consortium, collaboration

1. Transforming evaluation education in Canada
Evaluation education in Canada is going through a transformative experience through the successful establishment of
the Consortium of Universities for Evaluation Education (CUEE) - a voluntary association of post-secondary
institutions that is committed to expanding graduate-level evaluation courses and programs for current andprospective evaluation practitioners (CUEE, 2011a). CUEE is:●a collaborative partnership among universities, government and the Canadian Evaluation Society (CES) for the
purpose of building capacity in graduate-level education and training in Canada●a voluntary association of institutions and programs that is dedicated to expanding graduate level opportunities
for evaluation education and training across Canada●
part of a multi-organizational strategy to increase opportunities for current and prospective evaluation

Japanese Journal of Evaluation Studies, Vol.12, No.2, 2012, pp.11-22

12●

Keiko Kuji-Shikatani

James McDavid

J. Bradley Cousins

Heather Buchananprofessionals to acquire the knowledge and skills required to become credentialed evaluation practitioners.
(CUEE, 2011b)
providing learning opportunities for students/evaluators who are working towards the Canadian Evaluation
Society,s Credentialed Evaluator (CE) designation in fulfilling the requirement for education and training thatcan be satisfied by successfully completing a graduate-level credential in evaluation.

2. Need for quality evaluation professionals in Canada
Commissioned by the Federal Treasury Board Secretariat's (TBS) Centre of Excellence for Evaluation (CEE),
Cousins and Aubry (2006) conducted a study to identify actionable strategies and options for government to foster
advanced professional development for evaluators in the interests of enhancing quality assurance in the evaluation
function. As part of this study, Cousins and Aubry conducted a survey of university-based evaluation training
programs and options based on internet searches and telephone consultations and found that in Canada:●
Advanced level university training opportunities leading to credentials were limited (i.e., no degree programs inevaluation were available across the country; only three graduate certificate programs; one undergraduate
diploma program at a community college; graduate study in evaluation offered in universities are often limited
to course-level experiences; it is possible for students to complete thesis work in evaluation but only underdisciplinary programs such as education, public administration, and applied psychology; etc.)●
Evaluation training opportunities were more widely available (i.e., university courses in evaluation and related
topics exist on a broad basis, with several universities offering more than one evaluation course within faculties
or departments.).
In October, 2006, Treasury Board Secretariat sponsored a national meeting of the federal government, selected
universities, and the Canadian Evaluation Society to explore the interest of the university community in contributing
to the education/training of the next generation of evaluators.(CUEE, 2008) During this meeting the idea of a,
Consortium was first raised by university representatives to address the federal government s need to build capacity,
to educate more evaluators and the CES s interest in professionalizing evaluators in Canada.
A draft business plan for CUEE was discussed in the March 2008 TBS sponsored second national meeting topropose the creation of CUEE with the goal of increasing access to graduate credentials in Evaluation by creating andsustaining opportunities for students and practitioners (in Canada and eventually abroad) to take courses foruniversity credit that lead to graduate credentials in evaluation. This goal was intended to address the problems
identified:●A broad shortage of graduate level evaluation education and training in Canada, as well as barriers to accessbuilt into the Canadian academic system, both institutional and jurisdictional.
●
While graduate certificate and diploma programs were starting to be developed, more programs were needed,
and from a student perspective, there was much room for improvement in accessibility.
●
With few exceptions, universities and university colleges in Canada required students to apply for admission tothat institution (to Faculties of Graduate Studies for example) and to maintain an affiliation with that university
throughout their graduate education.
●
Although students could take coursework elsewhere and have those courses credited towards a credential intheir home institution, typically the courses that a student takes have to be approved in advance. In effect,

Consortium of Universities for Evaluation Education and its
Impacts on Professionalizing Evaluation in Canada

13

students have to negotiate the transfer of courses from other institutions to their home institution.
The new Policy on Evaluation in the Canadian Federal government increased the need for an expanded pool of
qualified evaluation professionals (TBS, 2009, 2011). The Policy mandated that all federal programs be evaluated ona five-year cyclical basis. Recognition of the need for the professionalization of evaluators influenced the,development of the CES s CE designation and the CUEE (McDavid & Cousins, 2010).

3. Partnership to offer graduate education for evaluation practitioners
CUEE began in the fall of 2008 as partnership among universities, the federal government and CES to develop
capacity in Canada to offer graduate education (courses and credentials) for prospective and current evaluation
practitioners.
The vision is that students and practitioners will be able to access courses and programs from one or more of theConsortium institutions and have their coursework credited towards a credential in their home institution.
Objectives The Consortium of Universities for Evaluation Education aims to:increase the number of qualified evaluators in Canada by providing education opportunities for evaluators
wanting to obtain the Canadian Evaluation Society,s Credentialed Evaluator designation;●build capacity among Canadian academic institutions to develop and offer advanced education/training for
professional evaluators;●
enhance students, access to high quality graduate level instruction in evaluation;●
support the development of graduate certificate and degree programs in evaluation by sharing course
syllabi, proposals for credentials and other materials;●
operate nationally;●
be self-sustaining; and●be governed by the academic institution partners, with input from other key partners.
●

Guiding Principles The Consortium of Universities for Evaluation Education is committed to:increasing access to quality graduate level instruction in evaluation through a mix of in-person course
offerings and internet-based (or distance mode) delivery mechanisms;●
providing pre-service and in-service professional development for evaluation practitioners that is informedby contemporary theory and professional practice;●ensuring shared access to programs and courses among Consortium members;●
focusing on students, needs;●
maintaining high academic standards as defined within respective provincial jurisdictions and member
universities;●
upholding the highest professional standards and ethics of professional practice in the field●
respecting evaluation as an interdisciplinary domain of inquiry;●
valuing the diversity of information needs of the evaluation community and contexts for practice●respecting Consortium members, interests and autonomy; and●responding to market trends and opportunities in government, voluntary, private and other sectors.
●

14

Keiko Kuji-Shikatani

James McDavid

J. Bradley Cousins

Heather Buchanan

CUEE has since revised its business plan, established the governance (CUEE, 2011c) structure that includes theConsortium Council (CUEE, 2011d), co-Chairs, the Operations Committee(CUEE, 2011e) and the Secretariat(CUEE, 2011f) which is currently located at the University of Victoria, School of Public Administration.
Memberships are renewed annually in September and include Canadian chartered universities, university colleges
with an interest in developing and offering graduate level evaluation-related courses and credentials. TBS and CES
are ex officio members of the Consortium Council (CUEE 2011a) CUEE conducted a cross-Canada inventory of
university-based evaluation education (CUEE 2009) and updates that inventory on an ongoing basis. The Secretariat
hosts a bilingual website (http://www.evaluationeducation.ca/ ) that features member institutions, courses, programs
and more detailed information such as course syllabi to facilitate members, understanding of what is being offeredelsewhere in the Consortium and to facilitate building complementary course offerings. Access to the SharePoint
website is restricted to members only and is intended to facilitate information sharing and collaborations among
members.
CUEE Member institutions are encouraged to collaborate at several levels. These include information sharing course syllabi, program proposals, and other materials to learn about existing and proposed evaluation-related
educational opportunities to position their own initiatives so that they complement existing courses and programs,
development and delivery of courses (and course modules), and credentials. (CUEE 2011a)
Since CUEE was created, members have expanded their graduate evaluation offerings. Four of the thirteen(2011 memberships) Consortium members have developed or are developing online graduate certificates in
evaluation and these will be available across Canada. Having online credentials will facilitate collaboration among
CUEE partners.
In addition to working on projects to facilitate collaboration, CUEE has undertaken a project for the CEE tocross-walk CUEE member evaluation curricula to the draft competencies for federal evaluators that are being
developed to guide training for evaluators in the Federal Government of Canada. An additional and related project
has seen the development of a crosswalk between CUEE member curricula to the core competences for professional
evaluators that were developed by the CES as part of the Credentialed Evaluator professional designation project. A
third project has been undertaken to assess the need for financial support and mentorship opportunities for students
who graduate from CUEE member programs. This latter project is being conducted in collaboration with theCanadian Evaluation Society Education Fund (CESEF), which is committed to supporting students who wish to
pursue graduate education in evaluation in Canada.
One of the most unique endeavours is the CUEE Framework for Collaboration on Shared Inter-university
Graduate Evaluation Education Courses which will facilitate students from CUEE member institution accessing
courses from other CUEE member institution programs and gain credits readily from their own universities without
going through the more complicated procedures for individual pre-approval of credits gained from another postsecondary institution. Given that education in Canada is a provincial matter outside of federal jurisdiction, theagreement will also assist in harmonizing university level evaluation education across the country.
FRAMEWORK FOR COLLABORATION ON SHARED INTERUNIVERSITY GRADUATE EVALUATIONEDUCATION COURSES (based on the Canadian Association of Graduate Studies template)
PurposeThis Agreement is to provide students in good standing enrolled in a graduate degree or diploma programat a CAGS (Canadian Association of Graduate Studies) member university the opportunity to avail
themselves of courses offered at another member institution (host) for transfer credit to the program at their●Consortium of Universities for Evaluation Education and its
Impacts on Professionalizing Evaluation in Canada●

15

institution (home).
The definition of home is the institution in which the student applicant is enrolled and is expected toprovide the graduate degree or diploma. The host is defined as the institution at which course credits canbe obtained that can be counted toward a degree or diploma at the home institution. (CUEE 2011g)

For students this Framework addresses the limited availability of advanced-level university training
opportunities, and allows for evaluation training to be widely accessible in multiple disciplines through CUEE
institutional members across Canada, especially through online learning opportunities. For a vast and diverse country
as Canada, globally ranking the second largest in total area, CUEE offers exciting possibilities.

4. CUEE and the CES Credentialed Evaluator designation
In May 2010, CES launched the CES CE designation designed to define, recognize and promote the practice of
ethical, high quality and competent evaluation in Canada through a program for professional designations. The
designation means that the holder has provided evidence of the education and experience required by the CES to be acompetent evaluator. The designation is a service provided by CES to its members, who may elect to become
credentialed on a voluntary basis. It recognizes those with the education and experience to provide evaluation
services, and through its maintenance and renewal requirements, promotes continuous learning within our evaluation,
community. CES maintains a publicly available registry of CE s. CE is about providing a path for new evaluators
and a clearer direction for more established evaluators for their ongoing development. It is about allowing CEs to berecognized for what they have achieved and for their ongoing commitment to learning and improving their practice as
program evaluators. Qualification requirements for CE include:●
evidence of graduate level degree or certificate related to evaluation●
evidence of two (2) years (full-time equivalent) evaluation related work experience within the last ten (10)years.
●
education and/or experience related to 70% of the competencies in EACH of the five domains of Competencies
for Canadian Evaluation Practice (the list of competencies is available on the CES website,www.evaluationcanada.ca )●evidence of ongoing learning and professional development - 40 hours of learning every three years - required
for maintaining the CE designation (CES 2011a)
Critical elements of the CE designation include ethics, standards and competencies - all of which wereaddressed in the development of the CES Professional Designations Program (CES 2009). Competencies are theknowledge, skills, and dispositions program evaluators need to achieve standards of practice that constitute the basis
for sound evaluations (Stevahn, King, Ghere & Minnema, 2005). CES National Council noted that a well-structured
and agreed knowledge base is essential to any system of professional designation and proposed undertaking acrosswalk (cross referencing with the goal of determining points of overlap and difference) of different extant
knowledge bases in order to develop a comprehensive list of evaluator competencies. CES Professional
Designations Core Committee undertook a crosswalk of evaluator competencies building on the work of Establishing
Essential Competencies for Program Evaluators (Stevahn, King, Ghere & Minnema; 2005). The crosswalk informed
the development of the CES Competencies for Canadian Evaluation Practice by examining guidelines developed byorganizations that primarily function to advance the professional practice of program evaluation (CES 2008). These
included the 2002 version of TBS Competencies for Evaluators in the Government of Canada (as a major employer

16

Keiko Kuji-Shikatani

James McDavid

J. Bradley Cousins

Heather Buchananand purchaser of evaluation); the 2004 Joint Committee Program Evaluation Standards (adopted by CES National,
Council); the 2004 version of the American Evaluation Association s Guiding Principles for Evaluators; and theUnited Nations Competencies for Evaluators in the United Nations System (to provide a broader international
perspective). After Membership consultations, Town Hall meetings across Canada and Expert consultations, theCompetencies for Canadian Evaluation Practice were defined (CES 2010).
1.0 Reflective Practice competencies focus on the fundamental norms and values underlying evaluation
practice and awareness of one,s evaluation expertise and needs for growth.
1.1 Applies professional evaluation standards
1.2 Acts ethically and strives for integrity and honesty
1.3 Respects all stakeholders
1.4 Considers human rights and the public welfare in evaluation practice
1.5 Provides independent and impartial perspective
1.6 Aware of self as an evaluator (knowledge, skills, dispositions) and reflects on personal evaluation practice(competencies and areas for growth)
1.7 Pursues professional networks and self development to enhance evaluation practice
2.0 Technical Practice competencies focus on the specialized aspects of evaluation, such as design, datacollection, analysis, interpretation, and reporting.
2.1 Understands the knowledge base of evaluation (theories, models, types, methods and tools)
2.2 Specifies program theory
2.3 Determines the purpose for the evaluation
2.4 Determines program evaluability
2.5 Frames evaluation questions
2.6 Develops evaluation designs
2.7 Defines evaluation methods (quantitative, qualitative or mixed)
2.8 Identifies data sources
2.9 Develops reliable and valid measures/tools
2.10 Collects data
2.11 Assesses validity of data
2.12 Assesses reliability of data
2.13 Assesses trustworthiness of data
2.14 Analyzes and interprets data
2.15 Draws conclusions and makes recommendations
2.16 Reports evaluation findings and results
3.0 Situational Practice competencies focus on the application of evaluative thinking in analyzing and attending
to the unique interests, issues, and contextual circumstances in which evaluation skills are being applied.
3.1 Respects the uniqueness of the site
3.2 Examines organizational, political, community and social contexts
3.3 Identifies impacted stakeholders
3.4 Identifies the interests of all stakeholders
3.5 Serves the information needs of intended users

Consortium of Universities for Evaluation Education and its
Impacts on Professionalizing Evaluation in Canada

17

3.6 Attends to issues of evaluation use
3.7 Attends to issues of organizational and environmental change
3.8 Applies evaluation competencies to organization and program measurement challenges
3.9 Shares evaluation expertise
4.0 Management Practice competencies focus on the process of managing a project / evaluation, such as
budgeting, coordinating resources and supervising.
4.1 Defines work parameters, plans and agreements
4.2 Attends to issues of evaluation feasibility
4.3 Identifies required resources (human, financial and physical)
4.4 Monitors resources (human, financial and physical)
4.5 Coordinates and supervises others
4.6 Reports on progress and results
4.7 Identifies and mitigates problems / issues
5.0 Interpersonal Practice competencies focus on people skills, such as communication, negotiation, conflict
resolution, collaboration, and diversity.
5.1 Uses written communication skills and technologies
5.2 Uses verbal communication skills
5.3 Uses listening skills
5.4 Uses negotiation skills
5.5 Uses conflict resolution skills
5.6 Uses facilitation skills (group work)
5.7 Uses interpersonal skills (individual and teams)
5.8 Uses collaboration / partnering skills
5.9 Attends to issues of diversity and culture
5.10 Demonstrates professional credibility

CUEE members are encouraged to use the Competencies for Canadian Evaluation Practice as a foundation fordeveloping programs, courses, and syllabuses, in partnership with CES in expanding opportunities to acquire
graduate credentials in evaluation - a key part of becoming a Credentialed Evaluator. (CUEE 2009) Preliminary
findings from the CUEE-CES competencies crosswalk project suggest a high degree of overlap of CUEE member
curricula and the CES Competencies for Canadian Evaluators.

5. Examples of Graduate Curricula in the Consortium of Universities for Evaluation
Education
Among the Consortium members, there has been considerable development of new graduate credentials (Graduate
Certificates and Graduate Diplomas) in the last three years. We will summarize a recent online program that wasdeveloped in the past two years, a program that is offered on campus at the University of Ottawa in a collaborative
arrangement between the Faculty of Education and the Faculty of Social Sciences and an interesting case of bridging
evaluation professional development and academic education at Carleton University (Ottawa). These programs

18

Keiko Kuji-Shikatani

James McDavid

J. Bradley Cousins

Heather Buchanansuggest the range of delivery options and organizational arrangements for offering CUEE-related graduate
credentials.
5.1 The University of Victoria Graduate Certificate and Diploma in Evaluation
These two programs were developed between 2009 and 2010 and are now being offered in the School of Public
Administration at the University of Victoria. Both programs are offered entirely online, that is, all the courses areaccessible to students anywhere as long as they meet the admission requirements and have appropriate internet
connectivity.
The Graduate Certificate is a four course program that has three prerequisites (courses that must be taken before
students can enrol in the program): introductory micro economics, introductory statistics and introductory researchmethods. The program is structured so that students take one online course each semester, completing the program infour semesters. The courses are sequenced as follows: introduction to program evaluation and performancemeasurement, qualitative evaluation methods and cases, economic evaluation methods and cases, and topics inevaluation, performance measurement, performance management and knowledge management. The latter course iscurrently focused on evaluating complex programs and policies. All four courses also can be taken as electives by
students in the three Masters Programs in the School of Public Administration and it is possible for students to takethe Graduate Certificate in Evaluation concurrently with their Masters Program.
The students apply to the program through the Faculty of Graduate Studies at the University of Victoria and arereviewed by an Admissions Committee in the School of Public Administration. Currently, the maximum annual
intake for the program is ten students. Nearly all the students in the program already have Masters Degrees and mostare practitioners. They come from all across Canada (Newfoundland west to British Columbia).
Students wishing to obtain the Diploma in Evaluation must complete all four courses in the Certificate Program
and then complete a program evaluation project with an actual organizational client. The evaluation project issupervised by a faculty member in the School of Public Administration and upon completion the project is assessed
jointly by the client and an academic committee in the School.
5.2 University of Ottawa Graduate Diploma in Program Evaluation
This interdisciplinary five course program was offered for the first time in 2006. It is a joint program developed andmanaged by the Faculty of Education and the School of Psychology and is offered in English and French. Although
the program can be fast-tracked in 12 months, most students choose to complete it part time over two years. An upper
level undergraduate research methods course with statistics is the only prerequisite for the program outside of normal
admission standards for graduate programs. Students can be admitted on conditional basis by committing to complete
the equivalent of the research methods course in their first term of registration. The program attracts a mix of inservice (practicing evaluators working full time) and pre-service (in-experienced evaluators choosing evaluation as acareer path).
The sequence of coursers is as follows: (i) All students begin in the fall term with a course on methods andpractice in program evaluation in which they plan an authentic evaluation as their major assignment. The course canbe in education or psychology, English or French. (ii) Students then implement the evaluation in the winter termeither immediately following the methods and practice course or one year later (depending on exigencies of the
program in question). This is done as part of a field studies course and is supervised by the instructor. Also offered in
the winter is (iii) a course on theory and contemporary issues in evaluation which can be taken in the first or second
year of the program. (iv) An elective course in research methods (quantitative or qualitative), planned change,organizational theory, or some other relevant choice is taken in the first spring/summer or, more likely, in the secondConsortium of Universities for Evaluation Education and its
Impacts on Professionalizing Evaluation in Canada

19

fall of the program. Finally, an integration (capstone) course is taken in the final spring/summer term. In that course
students produce a 25 page paper that addresses a practical problem in evaluation and develops strategies andsolutions on the basis of material covered in the program.
The annual intake for the program is about 20 students (15 English and 5 French speaking). While most courses
are offered on a face-to-face basis several make use of online resources or may in fact be hybrid or blended courses
with multiple modules being offered online. Currently one of the methods and practice courses is being offeredcompletely online and available to eligible participants outside of the Ottawa area. The program has a strong social
sciences emphasis with ties to educational evaluation and community psychology but it has been developed in such a
way that it meets the needs of participants from public sector / governance sectors and non-profit organizations. The
program is unique in that it is an inter-faculty entity which naturally augments its interdisciplinary character.
5.3 Carleton University Diploma in Public Policy and Program Evaluation (DPE) and International Programfor Development Evaluation (IPDET) in partnership with the Independent Evaluation Group of the WorldBank (IEG).
Another member of CUEE, Carleton University created its Diploma in Public Policy and Program Evaluation in2007. The DPE comprises six courses (of approximately 40 hours of instruction each). Applications are through the
School of Public Policy and Administration and require a Bachelor,s degree with an average of B+ or better. There
are four required courses: Policy and Program Evaluation (Evaluation Theory and Practice), Research Design I(Qualitative Methods), Research Design II (Quantitative Methods) and Cases and Applications in Evaluation (PADM5424). Two optional courses are selected from one of the following areas: additional evaluation methodology or
management of evaluation or policy fields1. The program is designed in an executive format which allows cohorts ofstudents to complete the program in 15 months while working full time. The DPE attracts students from thegovernmental, private and non-profit sectors. While delivered as an in-person classroom program at present, there areplans to provide distance learning opportunities within the program.
IPDET (which started in 2001) is a collaborative program between the IEG of the World Bank and Carleton
University. It is a four week in-residence program offered annually on the Carleton campus. It attracts roughly 200
students per year from over 60countries around the world. IPDET has two distinct components: a two-week, full-timecore curriculum of 15 modules2, followed by two weeks of selected 2 or 3 day workshops delivered by internationallyrenowned evaluation experts.
With the creation of the DPE, Carleton University and the IEG developed an interesting bridge between thesetwo programs, which created a hybrid of professional development and academic approaches to evaluation learning.
Participants in IPDET may acquire up the equivalent of three courses toward the six course DPE during theirattendance at IPDET. The courses for which credit may be acquired are:●
Policy and Program Evaluation: requires completion of the two-week Core Course of IPDET and completion of
assignments set by School of Public Policy and Administration (SPPA) instructors●
Research Design I (Qualitative Methods): requires completion of designated workshops (see equivalency table)
and completion of assignments set by SPPA instructors●
Research Design II (Quantitative Methods): requires completion of designated workshops (see equivalency
table) and completion of assignments set by SPPA instructors
IPDET participants may do one or all three of these courses for credit while they are attending IPDET, but mustfirst, independently apply for admission to the DPE. The above stated DPE qualifications apply and admission to
IPDET does guarantee acceptance to the DPE.
Similar to many professional development events, IPDET is attendance based. To bridge the benchmark of20

Keiko Kuji-Shikatani

James McDavid

J. Bradley Cousins

Heather Buchananacademic requirements, special post-IPDET assignments are set and graded by SPPA faculty. A grade of at least B+
is required to obtain the credit.

Conclusions and Challenges,
CUEE s role in preparing new evaluators and providing ongoing development for the CEs through providing
graduate-level evaluation education is critical to the success of the ongoing promotion of the practice of ethical, highquality and competent evaluation in Canada for the betterment of society. To our knowledge, the scale of thecollaboration among Canadian evaluation-related academic institutions, governments and the CES, is unique. The
transformation has just begun, CUEE membership is expanding and with it increased quality and diversity(methodological and in fields of specialization) in the evaluation learning opportunities in the programs and courses
available among the CUEE-member institutions benefitting the evaluation community in Canada.
Collaboration encounters challenges in part because post-secondary institutions guard their autonomy
vigorously. For CUEE, it is necessary to identify the opportunities for collaboration (courses in partner institutions
that are appropriate as electives in other CUEE programs, for example) and work through the barriers to students
being able to freely take courses in CUEE institutions and know that they will be credited towards the program inwhich they are enrolled in their home institution. Post-secondary institutions typically have limits on the numbers or
proportions of courses in a credential that can be taken elsewhere and transferred into a given program. Fortunately
in Canada, there is an umbrella agreement among all universities in the four Western provinces (British Columbia,
Alberta, Saskatchewan and Manitoba) called the Western Deans of Graduate Studies Agreement that facilitates any
graduate student being able to take courses at another university in any of those four provinces, transfer thecompleted course (grades) into their home institution while only paying tuition to their home institution. CUEE hasdeveloped a framework modelled on the Western Deans Agreement and is currently conducting a pilot project among
members to develop regular ways for students to take and transfer courses among programs.
Realistically it will take several years to systematically reduce the barriers that currently exist among
institutions. Once these barriers have been reduced (in some cases it will take universities changing their regulations)
it will be possible for CUEE collectively to explore developing collaborative graduate degrees in evaluation-related
fields.

Notes,
1 Elective courses for the DPE can be selected from the Master s program and have included Strategic Policy Evaluation, Cost
Benefit Analysis, etc.
2 IPDET Core program follows The Road to Results: Designing and Conducting Effective Development Evaluations, coauthored by IPDET founders Linda G. Morra Imas and Ray C. Rist, and is supplemented with small group work on actual
evaluation cases, round tables discussions and guest lecturers.

References
Buchanan, H., Maicher, B., and Kuji-Shikatani (2008, June 2) Canadian Evaluation Society Professional Designations Program
conference presentation at the Canadian Evaluation Society 2008 Ottawa National Conference retrieved November 26, 2011,
from Canadian Evaluation Society website

Consortium of Universities for Evaluation Education and its
Impacts on Professionalizing Evaluation in Canada

21

http://www.evaluationcanada.ca/txt/20090602̲designations.pdf
Canadian Evaluation Society (2011a) Professional Designations Program: What Qualifications are required? retrieved November
26, 2011, from Canadian Evaluation Society website
http://www.evaluationcanada.ca/site.cgi?s=5&ss=13&̲lang=EN
Canadian Evaluation Society (2010) Competencies for Canadian Evaluation Practice retrieved November 26, 2011 from the
Canadian Evaluation Society website
http://www.evaluationcanada.ca/txt/2̲competencies̲cdn̲evaluation̲practice.pdf
Canadian Evaluation Society (2009) CES Professional Designations Program Archives retrieved November 26, 2011 from the
Canadian Evaluation Society website
http://www.evaluationcanada.ca/site.cgi?en:5:61
Canadian Evaluation Society (2008) Crosswalk of Evaluator Competencies Version 10 retrieved November 26, 2011 from the
Canadian Evaluation Society website
http://www.evaluationcanada.ca/distribution/20080312̲ces̲professional̲designation̲core̲committee.pdf
Consortium of Universities for Evaluation Education (2011a, January 25) Membership Policy retrieved November 26, 2011, from
Consortium of Universities for Evaluation Education (CUEE) website:
http://www.evaluationeducation.ca/documents/Membership%20Policy.pdf
Consortium of Universities for Evaluation Education (2011b) About us retrieved November 26, 2011, from Consortium of
Universities for Evaluation Education (CUEE) website
http://www.evaluationeducation.ca/index.html
Consortium of Universities for Evaluation Education (2011c) CUEE Governance Policy retrieved November 26, 2011 from
Consortium of Universities for Evaluation Education (CUEE) website
http://www.evaluationeducation.ca/documents/Governance%20Policy.pdf
Consortium of Universities for Evaluation Education (2011d) Consortium Council retrieved November 26, 2011 from
Consortium of Universities for Evaluation Education (CUEE) website
http://www.evaluationeducation.ca/governance.html
Consortium of Universities for Evaluation Education (2011e) Operations Committee retrieved November 26, 2011 from
Consortium of Universities for Evaluation Education (CUEE) website
http://www.evaluationeducation.ca/governance.html
Consortium of Universities for Evaluation Education (2011f) Secretariat retrieved November 26, 2011 from Consortium of
Universities for Evaluation Education (CUEE) website
http://www.evaluationeducation.ca/governance.html
Consortium of Universities for Evaluation Education (2011g) Framework for Collaboration on Shared Inter-university Graduate

Evaluation Education Courses retrieved November 26, 2011 from Consortium of Universities for Evaluation Education(CUEE) website
http://www.evaluationeducation.ca/documents/Framework%20for%20Collaboration.pdf
Consortium of Universities for Evaluation Education (2009, June) Consortium of Universities for Evaluation Education (CUEE)

Project: Research on Evaluation Education at the Graduate Level in Canadian Universities - Final Report retrieved November
26, 2011 from Consortium of Universities for Evaluation Education (CUEE) website
http://www.evaluationeducation.ca/documents/evaluation%20education%20canada.pdf
Consortium of Universities for Evaluation Education (2008, Dec 19) Business Plan for the development of the Consortium of

Universities for Evaluation Education retrieved November 26, 2011, from Consortium of Universities for Evaluation Education(CUEE) website
http://www.evaluationeducation.ca/documents/cuee%20business%20plan.pdf

22

Keiko Kuji-Shikatani

James McDavid

J. Bradley Cousins

Heather Buchanan

Cousins, J. B., & Aubry, T. (2006, April). Roles for Government in Evaluation Quality Assurance: Discussion Paper Ottawa:
Centre of Excellence for Evaluation, Treasury Board of Canada, Secretariat. retrieved November 26, 2011 from Treasury
Board of Canada Secretariat website
http://www.tbs-sct.gc.ca/cee/tools-outils/qae-aqe/rgeqa-rgaqe-eng.asp
McDavid, J. & Cousins, J.B. (2010, May) Improving Access to Evaluation Education: The Consortium of Universities for

Evaluation Education (CUEE) conference presentation at the Canadian Evaluation Society 2010 Victoria National Conference
Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American
Journal of Evaluation, 26, 43-59.
Treasury Board of Canada Secretariat (2009) Policy on Evaluation retrieved December 18, 2011 from Treasury Board of Canada
Secretariat website http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024
Treasury Board of Canada Secretariat (2011) Evolution of Evaluation in the Federal Government in About the Centre of

Excellence for Evaluation retrieved December 18, 2011 from Treasury Board of Canada Secretariat website http://www.tbssct.gc.ca/cee/about-apropos-eng.asp(Accepted 15th October, 2012)23【Research Note】【】

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan– From the Perspectives of Civil Society Governance
Satoshi Morita
NPO Kokkyo naki Kodomotachi
morita̲s2@hotmail.com

Abstract
This article considers potentials and bottlenecks, promoting and hindering factors, and roles of civic organizations(COs) in introducing participatory evaluation (PE) to civil society, assuming PE as an effective means of active
citizen involvement in civil society in Japan. Major indicative findings: 1) PE can accelerate collaboration between
COs and other actors, and has merit for citizens with respect to accessibility to information, empowerment,
awareness-raising, etc.; 2) legitimacy is an important factor for promoting PE in civil society; and 3) COs are
expected to establish systems to support citizen participation in evaluation. Key issues for future consideration: 1)
establishing PE criteria and indicators depending on the nature of collaboration among actors is useful for its,,
application; and 2) the development of a tertiary sector built upon citizens further understanding of COs missions
and roles is incumbent to balancing the three sectors in achieving development of civil society, and to citizen
participation in decision-making in governing civil society.

Keywords
governance, collaboration, public interest, representation, legitimacy

Introduction
There are various ways for citizens to involve themselves in each of the actors that constitute civil society. For
example, involvement through collaborative relationships between government or private corporations (hereinafterreferred to as corporations ) and civic organizations (COs) is one way, while direct involvement with each actor isanother. Considering the fact that each actor plays its specific role in society, such participation mechanisms need tobe created that enable citizens to be involved with other actors while respecting their social roles.
This article considers, in a comprehensive manner, from the practical viewpoint, potentials of participatory
evaluation (PE) as well as ways of citizen participation, mainly focusing on the public sector, on the assumption that
PE is an effective means of active citizen involvement in civil society in Japan.
This article is structured as follows. Chapter 1 focuses on relationships between citizens and the major actors ofJapanese Journal of Evaluation Studies, Vol.12, No.2, 2012, pp.23-40 [Revised based on the translation fromJapanese Journal of Evaluation Studies Vol.11, No.1, pp.91-104(Japanese Version).]24

Satoshi Morita

the three sectors in society, based on the concepts of civil society, governance and collaboration that are crucial tobetter understanding the meaning of citizen participation. Chapter 2 overviews discussions on PE in the past.
Chapter 3 and 4 consider potentials in introducing PE, from the viewpoints of collaboration and direct citizen
participation respectively, taking into consideration the status quo of evaluation activities in the public sector. Chapter
5 discusses on promoting and hindering factors in spreading PE. Finally in conclusion, future prospects regarding theroles of PE and citizen involvement in civil society in Japan will be given.

1. Citizen Involvement with Major Actors in Civil Society
The emergence of COs in international society in recent years has drastically changed the perception of relationships
between the nation and COs. In Japan, the enactment of the law called Tokuteihieirikatsudo Sokushinho (the law topromote specified nonprofit activities), known as the NPO Law, and reforms of legal systems of public-service
corporations have made it possible to easily establish COs and thus, vitalized civil society since the late 90s. The
development of COs has led to active discussions on governance, mainly in Western countries since 1990. The
fundamental question here is how civil society should be governed, and by whom. Before further arguing this issue, it
is important to first overview the ways of relationships between major actors in civil society in clear distinction fromone another, and consider methods for citizens, who are considered the main constituent in democratic society, to beable to involve themselves actively and voluntarily in civil society as well as roles that COs can possibly take part in.
This Chapter reviews the concepts of civil society and governance and the roles of COs in civil society, anddiscusses the relationships between citizens and each actor, as these concepts are important for understanding various
ways of citizen participation in civil society.
1-1 Civil Society and Three Sectors
1-1-1 Concept of Civil Society
Many authors have considered that the concept of civil society is ambiguous and complex, and defined it in manydifferent frameworks, such as political and economic ideologies, doctrines and systems, nation-forming history andphilosophical ideas. Also, a number of authors have attempted to reframe civil society in relation to the raison-e＾tre
and roles of civil organizations for not more than 20 years. This article focuses, from the practical viewpoint, on theroles and functions of each of the major constituents in society, assuming that civil society is a society to beperceived from the viewpoint of citizens who are at its center as the main constituent.
1-1-2 Three Sectors in Society
A social system is considered to be composed of three sectors including the primary sector (or the public sector), thesecondary sector (or the private sector), and the tertiary sector (or the civic, voluntary, or non-profit sector). Figure 1
describes major entities in those sectors and classification criteria1. Government refers to governmental
institutions including the Cabinet, the Diet and administrative agencies, whereas Corporation accounts for private
entities that implement economic activities for seeking profit. Civic Organization applies to not-for-profit entities
being active under a certain sustainable system, run mainly by citizens who are engaged by voluntary choice andwithout compensation (see later), whereas Community explains where each individual is from or belongs to2.
Figure 2 shows the range of each sector and the position where each major actor stands in civil society from theviewpoint of citizens at its center. The term citizen here refers to collective individuals who do not belong to any
entity, or when they are released from any role they fulfill at the institution they belong to3.

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance

Figure 1
entity withorganization
structure(selfautonomous)

Major Entities in Three Sectors and Classification Criteria
Primary
Sectorrelevant
to the state

Government

irrelevantto the state

profit seeking

Corporation

not for profit(no return tomanager)

where thepersonbelongs

Source: The author, based on Salamon et al. (1996) and Jinno (2004)

Figure 2

25

voluntarywill

Secondary
SectorCommunity

Civic
Organization

Tertiary
SectorConceptual Model of Sectoral Classification and Major Actors in Civil Society
Tertiary Sector
Civil Society
Civic Organization

CitizenGovern-ment
Primary Sector

Corporation
Secondary Sector

Source: The author

1-2 Concepts of GovernanceThe development of argument about governance has led to the expansion of its concepts. Table 1 shows examples ofdefinitions and interpretations of various concepts of governance. Among these concepts, (a) good governance ismainly applied to the primary sector, whereas (b) corporate governance and (c) NGO governance are to the secondary
sector and (d) IT governance is to the tertiary sector. These four describe governance within internal organization ofeach major actor, focusing on a mechanism in which the major constituent of each sector actively involves itself indecision-making. It should be noted that the citizen, clientele and beneficiary, all of which enjoy services provided by26

Satoshi Morita

Table 1

Example of Definitions and Interpretations of Typical Governance Concepts

Governance ConceptsaGood Governanceb

Corporate Governancec

IT Governanced(Good) NGO Governancee

Public Governancef

Private Governanceg

Voluntary/Civic Governance

Examples of Definitions and Interpretations
Good governance refers to the management of government in a mannerthat is essentially free of abuse and corruption, and with due regard forthe rule of law (IMF 2007).
* The term is often applied mainly in the context of development
assistance for developing countries, but can be applied to any governments
including those of advanced countries.
The corporate governance structure specifies the distribution of rights andresponsibilities among the different participants in the organization – suchas the board, managers, shareholders and other stakeholders – and lays
down the rules and procedures for decision-making (ECB2004).
IT governance is the responsibility of the board of directors and executive
management…it consists of the leadership and organisational structuresand processes that ensure that the organisation’s IT sustains and extends
the organisation’s strategies and objectives (ITGI2003).
1) A transparent decision-making process in which the leadership of anonprofit organization, in an effective and accountable way, directs
resources and exercises power on the basis of shared values; and 2) GoodNGO governance is based on the distinction between organizational
entities (management and the governing body) and the distribution ofdecision-making power between them (Wyatt 2004).
* These definition can be also applied to NPOs (see later).
Public governance, whose authority is the State, government or public
sector, relates to the process by which a society organizes its affairs andmanages itself (UNDESA 2007).
Private governance is composed of the decision making processes and thebinding decisions of private groups that affect the quality of life andopportunities of a larger public (Rudder 2008).
The decision-making mechanism or structure of civic organizations that
affect the citizen or civil society (the author).

Note: NGO stands for non-governmental organization. Regarding the concept of (g), the author found very few references and thus, applied relevant
elements of the other two concepts of sector governance ((e) and (f)) to tertiary sector.
Source: The author

the major actors respectively, are not considered part of the main constituents of three sectors. With regard to (e)public governance, (f) private governance and (g) voluntary/civic governance, all of which explain governance ofeach entire sector, there is no common understandings or interpretations yet, as seen in the fact that not many relevant
references can be found.
Meanwhile, Yamamoto (2005) quotes, from Rhodes (1997) and other publications, a set of views on
governance focusing on the relationships between major actors in civil society, and calls them (h) network
governance, describing it as a complex concept. The argument behind the concept is that civil society is ruled throughcollaboration and coordination among major actors while each actor plays a complementary role to one another. He
also quotes from Peters (2000) the concept of (i) participatory governance based on the view that a mechanism ofwide citizen participation in policy-making is needed instead of the traditional hierarchical structure controlled bygovernment in the relationship between the primary sector and the citizen.
Table 2 shows a classification of those various governance concepts at different governance levels in threesectors. Based on the model in Figure 2, these concepts are considered all relevant to civil society governance, aConsideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance

Table 2

Classification of Concepts Relevant to Civil Society Governance

Governance
LevelsActor Level

Three
Sectors
Primary Sector

Good Governancea

Sector Levele

Society Level

Public GovernanceNetwork
GovernancehbCorporate Governancec

IT Governance

Secondary SectorfPrivate GovernanceParticipatory
Governancei
Tertiary Sector

27

NGO GovernancedgVoluntary/Civic
GovernanceSource: The author

concept of decision-making mechanism or structure in governing civil society. As mentioned earlier, the fourconcepts ((a) to (d)) describe governance within internal organization of each major actor, meanwhile the three ((e) to(g)) explain governance of each entire sector and the two ((h) and (i)) of entire civil society. Table 2 is visualized in
the two conceptual models in Figures 3 and 4. In Figure 3, the four ((a) to (d)) principally target internal systems andmechanisms of government, corporations, and COs, whereas the three ((e) to (g)) mainly focus on actions towards thecitizen by those three actors. In other words, the three ((e) to (g)) focus on the functions of each sector to influence
the external citizen as the main target to provide services to. This is different from internal governance as

Figure 3 Conceptual Model I − Major Actors ofThree Sectors and Governance Classification

Figure 4 Conceptual Model II − Major Actors ofThree Sectors and Governance Classification

Tertiary Sector

Tertiary Sector

Civic Organization

Civic Organization

d

h

h

i

g
Govern-ment

Primary
Sector
Source: The author

CitizenCitizenCorpo-ration

e
f
a

b

c

i

i
Corpo
-rationGovern-ment
Secondary
SectorPrimary
Sector
Source: The authorhSecondary
Sector28

Satoshi Morita

described in the four ((a) to (d)). In Figure 4, (h) shows networks on the basis of interactions between sectors,
whereas (i) refers to the application of citizen participation to each sector on the assumption that various stakeholders
take part in the management of civil society. Understanding these governance concepts is indispensable for further
considering various ways in which each actor relates to one another in civil society and the way civil society is.
1-3 Roles of COs in Civil Society
1-3-1 Characteristics and Roles of COs
In civil society in Japan, NPOs and public-service corporations4 are leading the tertiary sector. This article mainly
deals with NPOs since new public-service corporations are still in transition to new systems.
According to Salamon et al. (1996), COs in international society share the following characteristics: 1)
organizations; 2) private; 3) not profit distribution; 4) self-governing; 5) voluntary; and 6) public interest. Amongthese, the issues of (3) and (6) are considered especially important and often discussed by many researchers5.
Regarding the NPO Law of Japan, it is pointed out that there are legal systems to secure these two characteristics. In
terms of the former, the term non-profit is part of the name of the law itself, and the law prescribes prohibiting the
distribution of any profit or surplus to stakeholders of NPOs including their board members and the staff. With regard
to the latter, the term public interest is not exactly defined in the law. However, the law prescribes that NPOs
should, as an object, contribute to extending benefits to unspecific and numerous persons on the assumption that,
NPOs areas of activities fall in specified areas by the law such as international cooperation, environmental
preservation, and disaster relief. Accordingly, public interest is considered important6.
Yamaoka (2004) points out that COs can play a variety of roles in civil society, such as: 1) surveillance orobserving society; 2) social representative; 3) service provider; and 4) mediator as a third party. In other words, COs
are considered to be entities who can implement non-profit activities or activities of public interest, and who caninteract with each stakeholder from a more flexible standpoint, which distinguishes themselves from government or
corporations.
1-3-2 Concept of Collaboration,
With regard to governance of civil society, it is important to look at the way each actor s roles can complement to,others from the viewpoint of multidimensional management by major actors. The concept of collaboration is
crucial in considering relationships and ways of interactions of COs with government and corporations. For example,
Kanagawa (2008) quotes from Sullivan et al. (2002) three types of collaboration such as: 1) contract; 2) partnership;and 3) network. He also points out some characteristics of partnership as follows: 1) partnership involves
negotiation and co-making decision among various parties under mid-/long term common goals and thus, differs fromcontract; 2) partnership requires officially setting objectives and plans among relevant parties and thus, differsfrom network; 3) creation of added values that one single organization cannot achieve is an important element in
partnership.
1-3-3 Relationships of COs with Government and Corporations in Japan
Based on the above-mentioned points, this article overviews first the current situation of collaboration efforts
between COs and the local government in Japan. According to the survey results by the Cabinet Office (2004, 2007),the percentage of the local government implementing activities in collaboration with COs at the prefecture andmunicipality levels is high, and the percentage of NPOs working in collaboration or partnership with government isalso high7. That is, both COs and government are eagerly involved in establishing collaboration with each other.

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance29

Meanwhile, it is incumbent to pay attention to the fact that many NPOs in general seek from government anunderstanding of the significance of establishing equal relationships (Cabinet Office 2004, 2007). Here, the way
partnerships are built between COs and government is an important issue to be considered.
With regard to the relationships between COs and corporations, a larger number of corporations are actively
involved in corporate social responsibility (CSR) activities in recent years 8 and providing CSR services incollaboration with NPOs. In the meantime, both COs and corporations are still groping for a way of establishing
better collaboration towards the future9. Collaboration of COs with corporations differs in nature to a great degreefrom that with government. Many common values are shared under the goal of expanding public interest in the
primary and tertiary sectors, and there are a number of cases in which government provides public services throughfunding COs, whereas corporations are usually assumed not to outsource profitable activities to COs for profitmaking and commercial values (Sullivan et al. 2002, Kanagawa 2008).
1-4 The Citizen and Civil Society: Involvement in Three Sectors
1-4-1 Relationships of the Citizen with Each Major Actor
Next, this article considers relationships of the citizen with each actor in civil society in Japan from the viewpoint ofrepresentation−that is, who can represent and speak for the citizen, on the assumption, from the practical viewpoint,that the citizen is an independent entity of other actors as seen in Figure 2.
In government, the major actor of the primary sector, politicians are elected by the citizen through election
systems and thus, represent the citizen. In this case, legal and political representation is ensured. Meanwhile, officials
at administrative agencies who take charge of implementing policies, programs and projects are not elected by the
citizen and thus, only function to provide services of public interest without representing the citizen.
Likewise, COs, the major actor of the tertiary sector, do not represent the citizen, but can also function toprovide services of public interest. What distinguishes COs from government is that they can perform work while
declaring their own visions and strategies that reflect public interest and non-profitability10, not necessarily being,bound to government s policies and plans. This makes it possible for COs to implement activities that better meet theneeds of certain beneficiaries. In this regard, COs can speak for the citizen.
For corporations, the major actor of the secondary sector, profitability is the most important principle. There areoccasions when corporations meet some specific needs or respond to some specific preferences of the citizen as
customers or consumers, however corporations do not politically or legally represent the citizen, nor speak for the
citizen for the sake of public interest and in line with non-profitability. In summary, relationships of the citizen witheach actor are fundamentally all different in nature.
1-4-2 Citizen Involvement
In order for COs to be able to function to speak for the citizen, raising awareness of the citizen towards the raison-e＾treof COs in civil society is indispensable. Also, from the viewpoint of participatory governance, it is assumed
important for the citizen to be actively involved in each actor. Furthermore, citizen involvement is essential in thecollaboration of COs with government and corporations from the viewpoint of network governance. Taking thesepoints into consideration, this article mainly analyses the current situation of the primary sector mainly.
First, concrete means of active citizen involvement in government are secured in some legal systems that haverecently been enacted at the country level in Japan. For example, Gyoseitetsuzukiho, or the Administrative
Procedures Act (APA) took effect in 1994; Johokokaiho, or the Act on Access to Information Held by Administrative
Organs came into force in 2001; and Kankyoeikyohyokaho, or the Environmental Impact Assessment Law was put
into operation in 1999. These laws have increased opportunities for citizen participation and expanded the range of30

Satoshi Morita,
participants rights (Okubo 2005). Also, a number of ordinances aiming at urging citizen participation andcollaboration in administrative activities have recently been enacted at the municipality level11. In the meantime, there
are more gaps observed between municipalities that are more advanced in introducing citizen participation andcollaboration and those lagging behind − there are actually numerous municipalities that have not even considered
about enacting ordinances (Okubo 2005). Citizen participation in administrative activities is not common enough yet,and opportunities and means for individual citizens to be directly involved in government are still limited. This is also
true to the relationship between the citizen and corporations12.
,
As a matter of fact, from the citizen s standpoint, corporations are not entities that can speak for the citizen.
Also, very few citizens consider that public opinions are reflected on overall government policy13. Considering thecurrent situation in which there have been increasing cases of collaboration of COs with government andcorporations, establishing systems or mechanisms for the citizen to be actively involved with government or
corporations through collaborative relationships is desired in civil society for the future.
Next, as a means of active citizen involvement in NPOs, citizens can become shain, or regular members andattend the general meeting of members14. The NPO law prescribes that …provisions regarding acquisition and lossof qualifications for membership are not unreasonable, and there is no judgment for membership by the board of
directors. Therefore, there is no legal limit regarding membership and also no penal provisions for losing
membership. In other words, such legal systems are established that make it possible for the citizen as insiders toparticipate in decision-making of the management of NPOs.
Figure 5 shows a conceptual model demonstrating the ways citizens involve themselves with government,
corporations and COs in civil society, based on above-mentioned points. In the figure, (c) refers to the collaborative
relationship between government and corporations that already has been in operation for a long time, whereas (a) and(b) describe relatively new collaborative relationships of COs with government and corporations respectively. Also,(d), (e) and (h) show already existing ideas of involvement of the citizen with government, corporations and COs
respectively, while (f) and (g) represent new types of involvement of the citizen through the above-mentioned,
Figure 5 Conceptual Model of the Citizen s Involvement with Government, Corporation and Civic Organization
new collaboration
existing collaboration
new involvement
existing involvement

Tertiary Sector

Civic Organization

a

h
f

b
g

CitizendeCorporation

Govern-ment
Primary Sector

Source: The authorcSecondary Sector

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance31

collaborative relationships of COs with government and corporations respectively. Based on Figure 5, potentials of
PE in civil society will be considered in the following chapters.

2. Overview of Participatory Evaluation (PE)
2-1 Streams of PE
According to Cousins et al. (1998), although the term participatory evaluation (PE) has been used in different
contexts by different authors, there are two principal streams in PE. One is practical participatory evaluation (P-PE)
focusing on the function of evaluation use. This stream has been formulated through the application of PE todecision-making or problem-solving in implementing social programs, developing plans and measures, andorganization management in community, private corporations, schools, etc. in the United States and Canada since the1960s. The other is transformative participatory evaluation (T-PE) focusing on the empowerment of the sociallyvulnerable and achievement of social-justice which are part of an important process of social reforms. This stream
has been developed through the application of PE in the context of international development assistance indeveloping countries including Latin America and Africa since the 1960s.
2-2 Discussions on PE in Japan
It was not until 2000 that PE came to draw more attention and be often discussed in Japan. JICA (2001), for example,
considers potentials of introducing PE to the organization through reviewing methods and practices of PE in thecontexts of development assistance and international cooperation. Yamaya (2002) discusses PE in the contexts ofpolicy evaluation and administrative evaluation (see later). Miyoshi et al. (2001) argues PE in a comprehensive way,
applying case studies and statistic data, in relation to development assistance or policy evaluation and administrativeevaluation. According to Miyoshi et al. (2001), PE generally refers to an approach that allows various stakeholders toactively participate in evaluation, principally with an emphasis on influencing stakeholders in the entire evaluation
processes and improving on-going activities and projects as evaluation targets and thus, differs to a great degree in its
objectives from conventional evaluation focusing on value judgment of meaningfulness or effectiveness of activities
and projects. Miyoshi et al. (2001) explains that conventional evaluation mainly emphasizes ensuring accountability
to fund providers and acquiring knowledge and information required for making decisions on whether to implement
projects or for gaining lessons and making propositions, whereas PE basically focuses on building capacities andpromoting ownership of, and sharing knowledge among those implementing projects and beneficiaries who
participate in evaluation.

3. Current Situation and Future Potentials of Introducing PE: from the Perspectives ofCollaboration
This chapter considers potentials of PE, recognizing the fact that collaboration of COs with government andcorporations is especially important for accelerating active citizen involvement in governing civil society. The author
assumes that introducing PE in collaboration approaches is effective from the two perspectives: 1) PE as a means tofurther accelerate active involvement of COs with other major actors; and 2) PE as a method for evaluating, withcitizen involvement, collaboration efforts between actors. These two perspectives are not alternative to but can becomplementary to each other. The former is applied to (a) and (b), while the latter to (f) and (g) in Figure 5.

32

Satoshi Morita

Theoretical grounds and supportive evidences for these two suppositions are provided in the following sections,based on the model of stages in evaluation process proposed by Iwabuchi (2005) (see later) and the current situationof collaboration efforts between COs and government in Japan, focusing on the characteristics of PE described by
Cousins et al. (1998) as well as the differences between PE and conventional evaluation explained by Miyoshi et al.
(2001).
3-1 PE as a Means of Collaboration Approach (a P-PE Model)
The first supposition is that introducing PE as a means to accelerate active involvement of COs with other actors ((a)
and (b) in Figure 5) is effective. In general, the level of involvement of COs depends on the stage of evaluation
process, and ways of involvement differ in nature accordingly. Especially in PE, opinions of COs tend to be moreeasily reflected at the stage of value-judgment and decision-making because the degree of their autonomy becomes
higher compared to other stages of evaluation process. This characteristic provides grounds for the above-mentioned
supposition which derives from the stream of P-PE.
This characteristic will be further explained as follows. Iwabuchi (2005) describes that a general evaluation
process is classified into four stages along the timeline, such as: 1) evaluation design; 2) data collection; 3) data
analysis; and 4) judgment/decision-making. He also discusses the relationships between ways of involvement ofNPOs as a third party in evaluation activities at government level and the degree of independence of NPOs at eachstage of evaluation. Figure 6 is a conceptual model by the author, demonstrating an analytical framework based on
his discussions. According to him, autonomous involvement is a way of involvement with more independence,
whereas heteronomous involvement is with less independence such as outsourcing to NPOs by government. The
latter refers to passive involvement in evaluation activities. At the design stage, for example, NPOs often function,
as a contractor and government as an organizer and thus, the level of NPOs dependence is higher, whereas at the
judgment stage, it is assumed there is almost no occasion in which NPOs take part in evaluation in a dependentway. Meanwhile, the former refers to active involvement in evaluation activities. At the design stage, government

Figure 6

A Conceptual Model of Independence Level Depending on Ways of Involvement of a
Third Party and Stages of Evaluation Process

Level of Independence(passive/heteronomous involvement)

Dependence
Independence(active/autonomous involvement)
Evaluation
DesignData
Collection

Source: The author, based on Iwabuchi (2005)

Data
AnalysisJudgment/
Decision-making

Evaluation
Process

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance33

almost always makes decisions in light of internal rules, systems and budget and thus, again there is hardly an
instance in which NPOs take part in evaluation in an independent way, whereas at the judgment stage, NPOs are
assumed to participate in the entire value-judgment process. In summary, based on Iwabuchi (2005), the level of,NPOs dependence is higher at the earlier stage while the level of independence can become higher at later stages.
The author considers that the model in Figure 6 is relevant in discussing the potentials of introducing PE togovernment by COs. Conventional evaluation in a form of outsourcing to COs, for example, is deemed typical and away of passive collaboration in evaluation between government and COs, whereas PE can be a way of more active
collaboration in which COs involve themselves more independently. The application of PE enables COs to take partin the process of decision-making and value-judgment more actively than in the process of evaluation design as
described in Figure 6 and thus, PE is considered an effective tool for reflecting opinions of COs on value-judgment inevaluation and decision-making at government.
3-2 PE as a Means to Evaluate Collaboration Efforts (a T-PE Model)
The second supposition is that introducing PE as a method for evaluating, with citizen involvement, already existing
collaboration efforts of COs with government or corporations is effective. As described in the previous chapter, PE
has many characteristics that are useful for accelerating active citizen involvement with government or corporations((f) and (g) in Figure 5) through collaboration efforts of COs with other two actors ((a) and (b) in Figure 5), of whichcitizens are the beneficiaries. In addition to capacity building, fostering ownership and sharing knowledge, as
mentioned earlier, PE has merit for citizens also with respect to accessibility to information, awareness-raising,empowerment, fostering leadership, etc. These characteristics provide grounds for the above-mentioned supposition
which derives from the stream of T-PE. In concrete, citizens will have accessibility to wider varieties and larger
amount of information by participating in evaluation activities. Also, through dialogues with other major actors,
citizens will have more opportunities for gaining consciousness or awareness of themselves as citizen incomparative relationships with others in civil society, as well as for recognizing their roles and developing theircapacities. These merits can greatly contribute to accelerating citizen participation in governing civil society.
As a matter of fact, there seems to be enough room for applying PE involving citizens, to collaboration efforts
between COs and major actors. This is because each of those actors is already aware of the importance of evaluating
collaboration efforts, although in many cases actors are still at an early stage of introducing evaluation or evenconsidering appropriate approaches for applying evaluation. With regard to already existing collaboration efforts,
each actor recognizes the necessity of evaluating its own collaboration activities. In the case of collaboration between
COs and administrative organs, one of the requests to administrative organs from COs identified in the abovementioned survey results of the Cabinet Office (2007) was the necessity of developing mechanisms for evaluating
outcomes of partnership or collaboration between the two actors. On the other hand, the results of survey conducted
earlier by the Cabinet Office in 2004 show that among 40 respondent prefectures three had already establishedevaluation systems for collaboration activities and 21 were going to establish such systems. In other words, 24
prefectures recognize the importance of evaluation of collaboration activities. Among them, 20 prefectures recognize
the necessity of evaluating outcomes and processes of collaboration activities. Also, 18 prefectures attach importanceto internal evaluation, whereas 11 incline towards external evaluation.
In the case of collaboration between COs and corporations, one of the requests to COs from corporations
identified in the survey results of Fukuoka Prefecture (2009) was the necessity of reporting outcomes of collaboration
efforts in an objective manner.
When citizens participate in evaluation of collaboration activities, there are many important roles that COs canplay, such as providing knowledge of evaluation target projects and expertise of project evaluation in general,

34

Satoshi Morita

coordination between citizens as beneficiaries and other stakeholders, conducting public relations for civil society
from the perspectives of ensuring public interest and transparency, declaring visions of envisaged civil society for the
future from the standpoint of assuming accountability for civil society15 or speaking for the citizen, etc. It is expectedthat COs establish supporting systems for citizens participating in evaluation by performing these functions with theirexpertise or through their own networks, since government/administrative organs or corporations cannot play theseroles.
3-3 Current Situation of COs Participation in Evaluation at Administrative Organs
Based on the above-mentioned points, recent new evaluation systems in the government need to be drawn upon when
considering potentials of introducing PE for urging independent involvement of COs in existing evaluation,
frameworks in the government. In April 2002, as part of Japan s administrative reforms, the law called gyoseikikan
ga okonau seisaku no hyoku ni kansuru houritsu (the law regarding evaluation of policies undertaken byadministrative organs), known as the policy evaluation law, was enacted for administrative organs, as the main entity,
to make objective judgments on the effects of policies (in a wide sense) that are composed of three tiers, namely
policy (in a narrow sense),
program, and project, and provide information contributing to policy
formulation, and the system commonly referred to as seisaku hyoka (policy evaluation) was introduced to all 12
ministries and government offices. Prior to that, since the mid-90s, in Japanese municipalities, evaluations targeting
their own projects and activities have been implemented under the name of gyosei hyoka (administrative evaluation)and gradually become widespread16.
With regard to these evaluation activities at administrative organs, Miyoshi et al. (2001) points out that there isa strong applicability of PE methods to policy evaluation and administrative evaluation because both of them inprinciple take a form of self-assessment, and that these two evaluation systems can be made more effective throughthe introduction of PE. In fact, in terms of the two suppositions proposed in this article, the author considers it
possible to examine the effectiveness of applying PE to collaboration efforts between COs and government byanalyzing cases of such evaluations.
In the meantime, many authors point out that there is virtually no case or example of COs involvement in policy
evaluation or administrative evaluation, or applying PE based on collaborative relationships between COs andadministrative organs (Moteki 2006, Yamaya 2002). It is necessary to keep an eye on future developments regarding
the introduction of PE to administrative organs.

4. Potentials of Introducing PE−from the Perspectives of Direct Citizen Involvement
In continuation, this chapter overviews the current situation of introducing PE as a means to accelerate direct citizen
involvement with major actors in civil society ((d), (e) and (f) in Figure 5), especially with a focus on the primarysector, following the discussions on potentials of PE in relation to collaboration among those actors. According to theresults of survey targeting all 1,845 municipalities in Japan on the current situation of citizen participation in abovementioned administrative evaluation, conducted by Mitsubishi Research Institute (MRI) in 2009, among 845
municipalities which count for 46 percent of the total: 1) those who responded that citizens directly take part inestablishing systems for administrative evaluation account for about one percent at both policy and program levels; 2),those who responded that citizens voices were reflected on the choice of evaluation indicators, for example, accountfor less than five percent at both policy and program levels; and 3) those who responded that citizen representatives
take part in evaluation account for about six and 15 percent at the policy and program levels respectively (MRI 2009).

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance

35,
The municipality personnel mentions some reasons for all these low percentages, according to MRI s report,such as: 1) in general, among the municipality personnel, there is very little understanding of the objective for
introducing administrative evaluation or of its systems, which in turn makes them feel that administrative evaluationis merely a routine work or an additional burden for them; 2) it is difficult for them to see the way evaluation results
are utilized or the effectiveness of utilizing those results; and 3) there is no standard evaluation method including
evaluation criteria at municipalities (MRI 2009). Under these circumstances on the ground, it is apparently not easy toraise motivation to accelerate citizen participation in administrative evaluation.
,
Also, according to MRI s report, municipalities that were expecting to receive feedback from citizens throughthe publication of evaluation results account for about 14 and 31 percent at the policy and program levels respectively,
and there were more numbers of municipalities that were not planning to reflect citizens voices on their activities. In
addition, the report draws upon the fact that in many municipalities, conventional evaluation is still commonly
implemented only internally, also pointing out the importance of reconsidering evaluation approaches and thepossibility of citizen involvement (MRI 2009).
Next, it is pointed out that there is no system established for citizen participation in policy evaluation. Yamaya(2002) mentions that policy evaluation was operated and managed under the control of administrative organs and that
there was only very limited room for citizen involvement until then. The reason, according to him, was that thedesigning and development of policy evaluation systems was led by the bureaucrats. He also mentions the necessityof restructuring policy evaluation systems from the viewpoints of citizens as clients and allowing them toindependently involve themselves in policy evaluation since they are the beneficiaries.
As Miyoshi et al . (2001) and other authors mention, the significance of introducing PE to,government/administrative organizations is to allow them to adopt citizens opinions and reflect their voices on
decision-making through developing systems for citizens to participate in evaluations of its own policies, programs or
projects. This is highly related to the issues of participatory governance of civil society. As many authors havealready mentioned, in order to achieve democracy in civil society, reflecting pluralistic values on its process isextremely important, and PE can be an effective tool in this respect. For example, Yamaya (2002) points out thatpolicy evaluation should not be regarded merely as a tool for improving the efficiency of administrative activities ordisclosure of information, and asserts that policy evaluation will come to function and sound governance systems will
be ensured only after incorporating elements for citizen participation, such as involvement and collaboration of
stakeholders, empowerment of citizens for acquiring expertise and knowledge, policy proposals by citizens, etc. In
any case, it is expected that citizen participation in existing evaluation systems at administrative organizations will be
accelerated.

5. Promoting and Hindering Factors in Spreading PE,
It is noted that citizens willingness of participation is very low in general, which is a hindering factor for widely
spreading PE. In relation to the primary sector, for example, according to the survey results by MRI (2009), quite afew municipalities responded that there was almost no feedback or response from citizens in general to the
publication of results of administrative evaluation. In terms of the tertiary sector, according to the results of public
opinion survey concerning NPOs targeting 3,000 persons over 20 years of age nationwide, conducted by the
Cabinet Office in 2005, those who had took part in NPOs activities account for only less than 10 percent, and those
who were unwilling to participate in such activities in the future account for up to 49 percent (Cabinet Office 2005).
Underlying this situation is the lack of awareness of citizens as a constituent of civil society or recognition of their

36

Satoshi Morita

potential roles in civil society.
Even if this issue is solved, there would be no easy progress in widely spreading PE if there is no systematic
framework to apply it. In this respect, it is pointed out that ensuring legitimacy, or legalizing PE is important. For
example, as mentioned earlier, in recent years a number of ordinances aiming at urging citizen participation andcollaboration in administrative activities have been enacted at the municipality level, and following these trends, it isassumed that ordinances prescribing principles and mechanisms of PE or concrete methods of citizen participation inadministrative evaluation will possibly be enacted. As a matter of fact, considering that evaluation activities havebeen vitalized at administrative organizations through the legislation of policy evaluation and administrativeevaluation, ensuring legitimacy is certainly a promoting factor for widely spreading PE since positive influences canbe exerted on the spreading approaches as well as the degree of citizen involvement.

Conclusion: Issues regarding PE for Future Consideration
This article considered the ways of citizen participation mainly at administrative organizations in the primary sector
based on two suppositions regarding the effectiveness of introducing PE. It will be beneficial to gain lessons andmake proposals through case studies and analyses of PE in order to examine the effectiveness of introducing PE in
the future. Meanwhile, as mentioned earlier, almost no reference or report can be found of cases of evaluation forcollaboration activities between major actors, or of citizen participation in policy evaluation or administrativeevaluation. Therefore, it is highly desirable that PE will be widely spread in each sector including at administrative
organizations and that a number of cases of PE will be reported in the future. Also, through case studies, it isexpected that: 1) the methodologies of PE will be developed and systems for its application will be established; and,in concrete, that 2) knowledge will be accumulated with regard to appropriate methodologies of PE and approaches
for introducing PE depending on the entity of project implementation, the type of projects and the way of
implementation systems.
When applying PE to collaboration activities of COs with government or corporations, it can be effective toestablish new evaluation criteria, indicators and guidelines depending on the nature of collaboration from theviewpoint of, for example, the degree of maturity of civic society. Accumulating know-how and experience throughdeveloping such systems will lead to accelerating citizen participation in various activities in entire civil society inJapan in the future. COs play crucial roles for citizen participation in evaluation activities, and thus, are expected toestablish good supporting systems as entities that can speak for the citizen. In the meantime, the raison-e＾tre and rolesof COs are generally not understood well in civil society in Japan, and it is urgently needed to promote further
understanding among major actors as well as citizens in order to accelerate citizen participation in civil society.
As for the side of the citizen, it is incumbent to clearly recognize the roles they can play in relation to COs, and,
it is expected that raising citizens awareness will revitalize COs and lead to the development of a tertiary sector,
which in turn will contribute to balancing the three sectors in achieving development of civil society. That is to say,sound, fair and better civil society will be realized, in the sense that citizens will be in an equal position to otheractors, through citizens, understanding of their rights and duties, opportunities and roles as constituents of civilsociety, as well as participating in important decision-making in governing civil society by utilizing approaches suchas PE.
The shortcomings of this article include the following: 1) it does not deal with public-service corporations orcommunities since it mainly focuses on NPOs among COs in Japan; 2) it does not discuss much about the situation
surrounding corporations; and 3) it does not deal with concrete cases of collaboration among actors or citizen

Consideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance37

participation since it emphasizes overviewing an entire picture of the current situation of citizen participation in civilsociety in Japan. It is highly expected that knowledge and experience will be accumulated through studies of concrete
cases of citizen participation in Japan by researchers and practitioners, research on different collaboration approaches
among major actors in civil society in other countries, and comparative studies of the current situation surrounding
the application of PE and citizen participation in civil society between various countries.

Acknowledgement
The viewpoints and thoughts covered in this article all come from the author. I would like to acknowledge thosereferees who provided me with instructive advice.

Notes
1 This article clearly distinguishes concepts such as citizen, civic organization, civil society and the tertiary sector, considering
the fact that these concepts are sometimes confused with one another and their relationships are not clarified in many
literatures.
2 This article focuses on the roles of COs and thus, does not deal with community.
3 In Figure 2,

citizenis independent of other three entities on the assumption that each individual who is a constituent of civilsociety can take different positions in different situations−that is, he/she becomesa (general) citizen

once he/she isreleased from any role to play at the institution he/she belong to, no matter what entity it is. In this article, the terms citizen
and civic organization are clearly distinguished.
4 Since the NPO Law came into effect in December 1998, the number of newly registered NPOs has been increasing. As of theend of August 2010, 40,689 NPOs are registered and certified (Cabinet Office 2010). Regarding public-service corporations,
legal system reforms were carried out in December 2008. Some authors point out that COs in Japan generally include
other types of institutions in addition to NPOs and public-service corporations. NIRA (2004) mentions thatShohiseikatsukyodokumiai (consumers, cooperatives), (Danchi)kanrikumiaihojin (home owners, association corporations),

Ninkachiendantai (certified territorial organizations), and Koekishintaku (charitable trusts) can be included in a wide variety ofCOs. Yamaoka (2004) refers to civil society organization (CSO) as private organization of various standpoints or,
characteristics, and explains that CSO also can include residents association such as neighborhood self-governing body orcommunity, as well as civic organization without juridical personality. However, this article does not deal with theseorganizations.
5 In the United States, (3) and 6) are considered indispensable and important characteristics, whereas in continental Europe, (3)is not necessarily indispensable and (6) is excluded (Kanagawa 2008).
6 NPOs receive a certification by the administrative agency in charge at the time of their establishment. NIRA (2004) points outthat not all NPOs are considered to be implementing activities of public interest because a high level of independence isensured in NPOs organization management after having received their certification. Also, many cases of decertification ofNPOs have recently been reported (Cabinet Office 2010).
7 According to the Cabinet Office (2004), 40 prefectural governments and 413 municipalities were implementing projects incollaboration with COs. Also, the Cabinet Office (2007) mentions that about 75 percent of 1,019 respondent NPOs among
3,000 NPOs as survey targets were being in collaboration or partnership with government.
8 According to the results of survey on CSR targeting member companies in 2009 by the Japan Business Federation, 74 percentof 437 respondent companies have stipulated CSR basic principles and strategies, appointed board members, and established38

Satoshi Morita

cross-company mechanisms as well as CSR-specific departments in order to promote CSR activities (Japan Business
Federation 2009).
9 Refer to Ishikawa Prefecture (2005), Fukuoka Prefecture (2009) and CIRAC (2009), for example.
10 Refer to Morita (2009), for example, for discussions of the roles of COs in establishing strategies and visions as a social
mission.
11 According to Okubo (2005), these ordinances are generally classified, based on the types of their provisions, as follows: 1)
basic principles for autonomy; 2) principles for participation and collaboration; 3) systematizing participation andcollaboration methods; 4) individual participation and collaboration systems; 5) provisions regarding support for NPOs; 6)
combining principles for participation and collaboration and provisions regarding support for NPOs; 7) provisions regarding
community organizations; and 8) participation and collaboration mechanisms for individual specific areas. Ordinances on theombudsman system, the public comment system and the council system are mainly classified into (3) and (4).
12 With regard to citizen involvement in corporations, Kojima (2007), for example, asserts that the citizen essentially has rights toexercise influence over corporations and refers to the concepts of right for reform and right for opposition as a direct
and an indirect right respectively. He explains that the latter is the citizen,s action itself that is not prescribed in law, while theformer has not been institutionalized yet although it needs to be legislated in the near future. Regarding CSR, it has not beenlegislated in Japan yet, and one main factor in the background is a strong opposition by the business circles against government
to take the lead in the legislation process (Japan Business Federation 2004).
13 According to the polls carried out by the Cabinet Office in 2010, more than 70 percent of the respondents indicated that public
opinions or ideas are not reflected on overall government policy (Cabinet Office 2010).
14 Shain here does not refer to the staff, but to members who have a right and a duty to vote at the general meeting of members,
the highest organ of decision-making at an NPO, that must be held at least once a year where decisions on important matters
are made (NPO C,s 2010).
15 Refer to Morita (2009), for example, for detailed discussions of accountability of COs.
16 As of October 2008, administrative evaluation has been introduced to 864 municipalities including all prefectural governments
and ordinance-designated cities among the all 1,857 municipalities in Japan (MOIAC 2009).

References
Cabinet Office of Japan (2004). Heisei 15 nendo naikakufu itakuchosa komyunityi saikou ni muketa kyodo no arikata ni kansuru

chosahoukokusho (Report of Survey on Collaboration Approaches towards Community Reconstruction - Consignment Survey
by the Cabinet Office of Japan, FY2003).
Cabinet Office of Japan (2007). Heisei 18 nendo shiminkatsudodantai kihonchosa houkokusho (Report of Basic Survey on Civil

Activities Organization, FY2006).
Cabinet Office of Japan (2005). NPO (minkanhieirisoshiki) ni kansuru yoronchosa (public opinion survey concerning NPOs).
Cabinet Office of Japan (2010). Shakaiishiki ni kansuru yoronchosa (Public Opinion Poll on Social Consciousness).
Cabinet Office of Japan (2010). Tokuteihieirikatsudosokushinho ni motoduku shinseijurisu oyobi ninshosu, funinshosu tou (The
Numbers of Applications, Certifications, and Decertifications based on the Law to Promote Specified Nonprofit Activities). In

NPO homupeji (NPO Homepage) . Retrieved on October 20, 2010 from URL: <http://www.npohomepage.go.jp/data/pref.html>.
Chubu Industrial and Regional Center (CIRAC) (2009). Kigyogawakaramita NPO tono kyodonikansuru,
chosakenkyujigyohokokusho (Report on Research Project regarding Collaboration with NPOs from the Corporations

viewpoint).
Cousins, Bradley J., and E. Whitmore (1998). Framing Participatory Evaluation. In E. Whitmore (Ed.), Understanding andConsideration of Potentials of Participatory Evaluation in Civil Society in Japan− From the Perspectives of Civil Society Governance39

Practicing Participatory Evaluation (New Directions for Evaluation No.80), San Francisco, CA, U.S.A: Jossey-Bass.
European Central Bank (ECB) (2004). Annual Report: 2004. Frankfurt, Germany.
Fukuoka Prefecture (2009). NPO to kigyo tono kyodojittai tyosahokokusyo (Report of Survey on the Actual Situation

surrounding Collaboration between NPOs and Corporations).
International Monetary Fund (IMF) (2007). Manual on Fiscal Transparency. Washington D.C., U.S.A.
Ishikawa Prefecture (2005). Kigyo to NPO no kyodonikansuru ikochosa (Survey on Plans for Collaboration between

Corporations and NPOs).
IT Governance Institute (ITGI) (2003). Board Briefing on IT Governance, 2nd Edition. Rolling Meadows, IL, U.S.A.
Iwabuchi, K. (2005). NPO Accountability Ivolved in Evaluation - A New Trend of Civic Participation at Municipalities. Journalof Public Policy Studies, 4, 71-82.
Japan Business Federation (2009). CSR ni kansuru anketochosakekka (Euquete Survey Results on CSR).
Japan Business Federation (2004). CSR heno taionitsuite (Responsive Actions towards CSR).
Japan International Cooperation Agency (JICA) (2001). Participatory Evaluation and International Cooperation.
Jinno, N. (2004). Dai 1 sho, dai 1 setsu: Atarashi shiminshakaino keisei - Kan kara min heno bunken (Section 1, Chapter 1:

Formulating New Civil Society - Decentralization of Power from Government to Nongovernment). In N. Jinno and Y. Sawai(Eds.), Sosharu gabanansu: Atarashii bunken, siminshakai no kozo (Social Governance - New Structure of DecentralizedPower and Civil Society). Tokyo: Toyo Keizai Inc.
Kanagawa, K. (2008). Kyodogata gabanansu to NPO - Igirisu no patonashippuseisaku wo jireitoshite (Collaborative Governanceand NPOs - Using the Case of the Partnership Policy in the United Kingdom). Koyo Shobo Corporation.
Kojima, H. (2007). Shiminshakai to koporeto gabanansu (Civil Society and Corporate Governance). Tokyo: Bunshindo
Publishing Corporation.
National Institute for Research Advancement (NIRA) (2004). Dai 3 sho, dai 1 setsu: Shuyona siminshakaisoshiki no genjo(Section 1, Chapter 3: Current Situation of Major Civil Society Organizations). In N. Jinno and Y. Sawai (Eds.), Sosharu
gabanansu: Atarashii bunken, siminshakai no kozo (Social Governance - New Structure of Decentralized Power and Civil
Society). Tokyo: Toyo Keizai Inc.
Ministry of Internal Affairs and Communications of Japan (MOIAC) (2009). Chihokokyodantai niokeru gyoseihyoka no torikumi
jyokyo - Heisei 20 nen 10 gatsu 1 nichi genzai (The Current Situation of the Implementation of Administrative Evaluation at
Municipalities - as of October 1, 2009). In Chihokokyodantai ni okeru gyoseikaikaku no torikuminitsuite (Regarding the StatusofAdministrative

Reformsat

Municipalities) .

Retrievedon

November

22,

2010

from

URL:

<http://www.soumu.go.jp/click/jyokyo̲20081001.html>.
Mitsubishi Research Institute (MRI) (2009). Chihojichitai niokeru gyoseihyoka nadoheno torikuminikansuru jittaichosa 2009 nen(Survey on the Current Situation surrounding the Implementation of Administrative Evaluation at Municipalities 2009).
Miyoshi, K. and Y. Tanaka (2001). Future of Participatory Evaluation - Concept and Utilization of Participatory Evaluation -. The
Japanese Journal of Evaluation Studies, 1(1), 65-79.
Morita, S. (2009). A Consideration of NGO Accountability in Japan - Accountability on Vision and Strategy, and a Conceptual

Classification of NGO Accountability. The Japanese Journal of Evaluation Studies, 9(2), 31-50.
Moteki, I. (2006). Kisotekijichitai ni okeru sankagatahyoka no tekiyo ni kansuru kenkyu - bunkengatashakai ni okeru

chihoukezararon no kokuhuku ni mukete - (Study on the Application of PE to Municipalities - Towards Overcoming the
Shortcomings of Municipalities Structure in Decentralized Society). 21 seiki shakaidezain kenkyu (Twenty-first Century Social
Design Studies), 5, 115-125.
,,
NPO Coalition for Legislation to Support Citizen s Organizations (C s) (2010). NPO hojin no unei (Management of NPOs). In

NPOWEB. Retrieved on November 22, 2010 from URL: <http://www.npoweb.jp/modules/faq/index.php?cat̲id=4>.
Okubo, N. (2005). The Current Situation and Future Prospects of Ordinances to Urge Citizen Participation and Collaboration.

40

Satoshi Morita

Journal of Public Policy Studies, 4, 24-37.
Peters, G. B. (2000). The Future of Reform. In G. B. Peters and D. J. Savoie (Eds.), Governance in the Twenty-First Century:

Revitalizing the Public Service. Montreal / Kingston, Canada: Mcgill-Queens Univ Press.
Rhodes, R. A. W. (1997). Understanding Governance: Policy Networks, Governance, Reflexivity and Accountability. Open
University. Buckingham, United Kingdom / Philadelphia, U.S.A.
Rudder, C. E. (2008). Private Governance as Public Policy: A Paradigmatic Shift. School of Public Policy, George Mason
University. Fairfax, VA, U.S.A.
Salamon, L. M., and H. K. Anheier (1996). The Emerging Nonprofit Sector: An Overview. Manchester University Pr.
Manchester, U.K.
Sullivan, H. and C. Skelcher (2002). Working across boundaries: Collaboration in public services, New York, NY, U.S.A:
Palgrave Macmillan.
United Nations Department of Economic and Social Affairs (UNDESA) (2007). Public Governance Indicators: A Literature

Review. New York, NY.
Wyatt, M. (2004). A Handbook of NGO Governance. The European Center for Not-for-Profit Law. Budapest, Hungary.
Yamamoto, H. (2005). Counter Relationship of Civil-Society State and the Concept of Governance. Journal of Public Policy
Studies, 5, 68-84.
Yamaoka, Y. (2004). Dai 3 sho, dai 4 setsu: Shiminkatsudodantai no yakuwarito kadai (Section 4, Chapter 3: Roles and Future

Prospects of Civil Activities Organizations). In N. Jinno and Y. Sawai (Eds.), Sosharu gabanansu: Atarashii bunken,
siminshakai no kozo (Social Governance - New Structure of Decentralized Power and Civil Society). Tokyo: Toyo Keizai Inc.
Yamaya, K. (2002). The Policy Evaluation and NPO - the other Experiment. Journal of Public Policy Studies, 2, 82-95.
(Accepted 15th June, 2012)

41

Publication Policy of the Japanese Journal of Evaluation Studies
Last revised on 15th February 2005
The Purpose and the Name
1．The Japan Evaluation Society (hereinafter referred to as “evaluation society”) publishes “The Japan
Journal of Evaluation Studies (hereinafter referred to as “evaluation study”) in order to widely releaseevaluation studies and outputs of practical activities to domestic and international academic societies,
interested individual and institutions, and contribute to the advancement and prevalence of evaluation
practice.
Editorial Board
2．The editorial board administrates editing of evaluation study based on the editorial policy stated below.
3．The editorial board is formed with less than 20 members of the evaluation society who are assigned by
the board of directors. Terms of editors are two years but can be extended.
4．The editorial board assigns one editor-in-chief, two vice-editors-in-chief, and a certain number ofstanding editors among the members.
5．The editorial board may hold at least one meeting to discuss the editing policy, plans of editorial board,and others.
6．The editorial board reports activities to the board of directors as needed and receives approval. Also it isrequired to report the progress of the past year and an activity plan for the following year at the annual
conference.
7．The editor-in-chief, the vice-editors-in-chief and the standing editors organize the standing committee andadministrate editing on a regular basis.
Editorial Policy
8．The evaluation study, as a principle, is published twice a year.
9．The evaluation study is printed on B5 paper, and either in Japanese or English.
10．Papers published in the evaluation study are categorized as five types;
10.1．Review
10.2．Article
10.3．Research note
10.4．Report
10.5．Others
11．The qualified contributors are members of the evaluation society (hereinafter referred to as “members”)
and persons whose contribution is requested by the standing editors. Joint submission of members andjoint submission of non-members with a member as the first author are accepted. Submission by theeditors is accepted.
12．Submitted manuscripts are treated as the above categories, however, the standing editors will decide
based on the application of the contributors and the following guidelines;
12.1．“Review” is a paper, which provides an overview of evaluation theory or practice. The editorial board
will make the decision regarding publication.
12.2．“Article” is considered as a significant academic contribution to the theoretical development ofevaluation or understanding of evaluation practice. The standing editors committee makes adoption
judgments following the referee-reading process described in the next section.
12.3．“Research note” is a discussion equivalent to the intermediate outputs of a theoretical or empiricalenquiry. The standing editors committee makes adoption judgments following the referee-reading
process described in the next section.

42

12.4．“Report” is the study report related to a practical evaluation project or evaluation. The standing editorscommittee makes adoption judgments following the referee-reading process described in the next
section.
12.5．“Others” includes requested papers for special editions organized by the editorial board andannouncements from editorial board to members regarding publication.
13．The editorial board selects two referee readers. For the “article”, the editorial board makes adoption
judgments referring to the results from referee readings and comments provided by one editor assigned by
the editorial board. For “review”, “research note”, “report” and “others”, the editorial board makes
adoption judgments referring to the results from referee readings.
14．When editors submit a manuscript, the editors are not allowed to attend any of the standing editorscommittee meetings or editorial board meetings regarding the manuscript.
15．The standing editors have alternative of approval or not-approval for adoption judgment of manuscripts
submitted to any categories. However exception is permitted if the editorial board approves the
publication after minor rewrite. Even if the manuscripts are considered insufficient as an “article”,
standing editors can decide whether the manuscripts are published as a “research note” or “report” if theauthors wish to publish.
Formulation and Release of Submission Procedure
16．The editorial board formulates the submission procedure based on the editorial policy described above
and release after approval from the board of directors.
Distribution
17．The evaluation study is distributed to all members for free and distributed to non-members for a charge.
Distribution of the Printed Manuscript
18．30 copies of the respective paper are reprinted and distributed to the authors. The authors must cover any
costs incurred by author,s requests for printing more than 30 copies.
Release on the Internet
19．The papers published in the evaluation study are released on the internet with approval from the authors.
Copyright
20．Copyright of papers which appear in the evaluation study is attributed to the respective authors. Editorial
right is attributed to the evaluation society.
Office21．The office is in charge of administrative works for editing, distribution, and accounting.

43

Information for Contributors(For English Papers)
Last revised on 29th September 2008
1．“The Japanese Journal of Evaluation Studies” is the publication for reviews, articles, research notes, andreports relating to evaluation.
2．“The Japanese Journal of Evaluation Studies” is primary published to provide opportunities for membersof the Japan Evaluation Society (hereinafter referred to as “members”) to exchange findings, and tocontribute to further development of the study of evaluation both domestically and internationally. As aprinciple, this journal publishes the contributions submitted by the members. With the exception ofrequested papers, the first author must be a member. A submission (as the first author) is limited to one
manuscript that has not been published or submitted in any form for another journal of academic
association etcetera.
3．Adoption judgments of the manuscript are made at the discretion of the editorial board. Comments fromtwo referee readers who are appointed for every manuscript are referred to in the screening process (theeditorial board requests referee readers without notifying the author of manuscript).
4．Payment for the manuscript is not provided.
5．Papers published in “The Japanese Journal of Evaluation Studies” are released on the Internet at
homepage of this academic society.
6．Regarding submission, manuscripts must be identified as one of the following categories: 1) article, 2)
review, 3) research note, 4) report, and 5) others. However, the final decision of the category is made by
the editorial board.
“Article” is considered as a significant academic contribution to the theoretical development of evaluationor understanding of evaluation practice.
“Review” is a paper which provides an overview of evaluation theory or practice.
“Research note” is a discussion equivalent to the intermediate outputs of a theoretical or empirical study
in the process of producing an “article”.
“Report” is the study report related to a practical evaluation project or evaluation.
“Others” are manuscripts for special editions requested by the editing committee.
7．Manuscript Submission(1) Manuscripts may be written in either Japanese or English.
(2) Correction by the author is only for the first correction.
(3) English manuscripts should be submitted only after the English has been checked by a native speaker.
(4) Submit four hard copies (A4 size) of the manuscript. Contact information including mailing address,
telephone number, fax number, and e-mail address, and the category of the manuscript should beclearly stated.
For approved manuscripts, after necessary rewriting, the author needs to submit two hard copies of the
final paper as well as a text file saved on a DOS/V formatted floppy disk. Original figures, charts, andmaps should be provided.
(5) Total printed pages should not exceed 14 pages. Any cost incurred by printing more than 14 pages
must be covered by the author.
(6) The layout for English papers should be 30 mm of margin at left and right side, 10pt for font size, 43

44lines on A4 paper (about 500 words per page). An abstract of 150 words should be attached to the front.
14 pages are equivalent to 7,000 words but the body should not exceed 6,000 words to allow for the title,
header, figure, chart, footnotes, and references. Please note that the number of pages may be more than
expected depending on the number of figures included.
8．Mailing address
Office of Japan Evaluation Society at International Development Center of Japan
Hitachi Soft Tower B 22nd Floor, 4-12-6, Higashi-Shinagawa, Shinagawa, Tokyo,
140-0002, Japan
Phone: +81-3-6718-5931, Facsimile: +81-3-6718-1651
E-mail: jes.info@idcj.or.jp

45

Writing Manual of the Japanese Journal of Evaluation Studies(For English Papers)
Revised on 18th September 2002
1．Text, Charts, Figures, Graphs, Diagrams, Notes, and References(1) The paper should be written in the follow order:First page: Title; the author,s name; Affiliation; E-mail address; Abstract (150 words); Keywords (5 words)
Second page: The main text; acknowledgement; notes; references(2) Section of the text should be as follow:
1.
1.1
1.1.1
1.1.2
(3) Source of the charts, figures, graphs, and diagrams should be clarified. Submitted charts and others will be
photoengraved, therefore it is important that the original chart is clear. Pictures shall be treated as figures.
Figure 1 Number of Students in the State of ○○Note:
Source:Table 1 Number of Accidents in the State of ○○Note:
Source:(4) Citation of literature in the text should be, (Abe 1995, p.36) or (Abe 1995).
(5) Note in the text should be, (------.1 )46(6) Note and references should be written all together in the end.
Note
1 --------.
2 --------.
(7) Reference should list the literature in alphabet order, and arranged in chronological order. Follow theexamples:Book: author (year of publication). Title of the book. Published location: publishing house.
(e.g.) Rossi, P. H. (1999). Evaluation: A Systematic Approach 6th edition. Beverly Hills, Calif: Sage
Publication.
Article from magazine: author (year of publication). Title. Title of the magazine, volume (number), pagepage.
(e.g.) Rossi, P. H. (1999). Measuring social judgments. American Journal of Evaluation, 15(2), 35-37.
Article in Book: author (year of publication). Title. In editor (Eds.), Title of the book. Published location:
publishing house, page-page.
(e.g.) DeMaio, T. J., and Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in thefield. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitiveand communicative processes in survey research. San Fransisco, Calif: Jossey-Bass, 177-196.
Book by two authors: surname, first name, and surname, first name. (year of publication). Title of the book.
Published location: publishing house.
(e.g.) Peters, T., and Waterman, R. (1982). In Search of Excellence: Lessons from America,s Best Run
Companies. New York: Harper & Row.
Book by more than three authors: surname, first name, surname, first name, and surname, first name. (year of
publication). Title of the book. Published location: publishing house.
(e.g.) Morley, E., Bryant, S. P., and Hatry, H. P. (2000). Comparative Performance Measurement.
Washignton: Urban Institute.
(note 1) If some references are from the same author with the same publication year, differentiate by adding
a,b,c as (1999a), (1999b).
(note 2) If the reference is more than a single line, each line from the second should be indented by threespaces.
(e.g.) DeMaio, T. J., and Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in thefield. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitiveand communicative processes in survey research. San Fransisco, Calif: Jossey-Bass, 177-196.

47

Referee-Reading Guideline
The Japanese Journal of Evaluation Studies Editorial Board,
The Japan Evaluation Society
Approved on 10th September 2005
1．Content of the Referee-Reading Guideline
This Referee-Reading Guideline is to provide explanation of the main publication judgment, procedure of the refereereading, to the members who submit the manuscript and for the members who are requested to conduct refereereading in order to carry out the procedure efficiently and effectively.

2．Purpose of Referee-Reading and the Responsibility of the Author
Referee-reading is necessary for the editorial board to make decisions of whether submitted manuscripts areappropriate to publish in the Japanese Journal of Evaluation Studies or not.
If there is doubt or obscurity identified in manuscripts during the referee-reading corrections may be required.
Therefore, referee-reading also contributes to the improvement of the submitted manuscripts. However, although themanuscripts are requested corrections, the author is still solely responsible in regards to the contents and it is not
attributed to the referee-readers.
Referee-readers are two persons who are requested by the editorial board depending on the specialty or the field of
the submitted manuscript. People who are not members of this academic society also may be requested.

3．Items of Consideration in Referee-Reading
Five points are considered in referee-reading, however, the importance of each may be different depending on thetype of manuscript.
(1) Importance and utility of the theme(2) Originality of the study(3) Structure of the logic(4) Validity of verification and methodology(5) Contribution to evaluation theory and practice-

For the article, all of above five are considered.
For the research note, especially (1), (2), (3), and (4) are considered.
For the report, especially (1), (3), and (5) are considered.
For the review, especially (3) and (5) are considered.

48

4．Attentions in submission of manuscript
Besides above five viewpoints, basic completeness as a paper is also considered, for example;- appearance of the paper is organized- written according to the writing manual- described simply and distinctive- verification data is appropriately used
- notes and references are corresponding with the text
- terminology is appropriately used
- no wording and grammatical mistakes- no errors and omission- no punctuation mistakes- expression in English abstract is appropriate- word count is according to the manual
The above mentioned forms and contents are also considered. There have been cases in which graduate students andpractitioners posted without organizing the manuscripts as a paper. On those occasions, referee-reading was notconducted. Necessary consultation is strongly recommended prior to submission.

5．Judgment Cases in Referee-Reading(1) In the case of the manuscript which is considered acceptable for the publication but is not yet complete:
The referee reader should evaluate carefully whether the paper can contribute to the development ofevaluation theory or evaluation studies.
-

Verification is lacking but the theory and formulation are useful for academic development.
Analysis lacking but useful for formation and promotion of new theory.
The literature review is not of a high standard but, the overall study is meaningful.
Comparative study is not up to standard but is meaningful as an example of application.
Analysis is lacking but it is meaningful as an evaluation of socially and historically important cases.
Analysis is lacking but it is meaningful as an evaluation of particular social activities.
Organization and expression are not up to standard as a paper but the contents are worthy to evaluate.
Logic is not strong enough but useful in practice.
The paper has significance as a report.

(2) In case of the manuscript which is considered as difficult for publication:- Awareness of the issue or setting of the problem is indecisive.
- Understanding or analytical framework of notion of basic terminology is indecisive or inappropriate.
- There is a lack in credibility of data for the grounds of an argument.
- There is no clear point of an argument or appropriateness of proof.
- Organization of the paper and presentation (terminology, citation, chart, etc) are inappropriate (or notconsistent).

49

6．Judgment
The final decision will be made on publication at the standing editors committee following one of four patterns (listed
below). However, these judgments are not based on the number of errors but on the strength of the overall report. In
the case of (3) and (4), there is a possibility to be published as a different type of paper. If it is published as a different
type of paper, major rewrite concerning the number of words may be required.
(1) The paper will be published.
(2) The paper will be published with minor rewrite.
(3) The paper will be published with major rewrite, however as a different type of paper (review, article,
research note, or report).
(4) The paper will not be published; however there is the possibility that it will be published as a different typeof paper (review, article, research note, or report).