Social Inclusion through Culture and the Arts: A Handbook for Evaluation

Social Inclusion throughCulture and the Arts:
A Handbook
for Evaluation
By The Agency for Cultural Affairs of Japan& Kyushu University Joint Research TeamThe Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team

Creating new values
for diverse individuals
in our society

S O C I A L I N C L U S I O N × C U LT U R E A N D T H E A R T S

Social Inclusion throughCulture and the Arts:
A Handbook
for Evaluation
The Agency for Cultural Affairs of Japan& Kyushu University Joint Research Team

Creating new values
for diverse individuals
in our society

Introduction

Contents

When you hear the word “evaluation”, you might picture interview
evaluations or personnel evaluations—a person in authority unilaterally
making decisions regardless of the feelings and intentions of the person
being evaluated. However, public grants, awards, and commissions should
not be evaluated by specific individuals, but by the diverse group of citizens
who make up the public. This handbook considers how in a democraticsociety, diverse citizens are the appreciators of public work and should
therefore be the true evaluators of socially inclusive cultural projects.
In English, the word “evaluation” means “to find the value of”, andemphasis is placed on discovering the value of something. Evaluation
should not only be done ex post facto to determine how effective an arts
activity was, but to build a better future. It is important that objectives areclarified and that evaluations are carried out in an appropriate manner using
standards that meet these objectives.
Put that way, evaluation may sound very difficult, but that is not the case. If
you read this handbook, we think you will agree that evaluation issomething you can do yourself.

Mia Nakamura

Lead Researcher, the Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team
Associate Professor, Faculty of Design at Kyushu University

2

Introduction

2

Digest4

What are socially inclusive cultural projects?

13

Before evaluation

29

Evaluation wisdom in practice39

Communication through evaluation

55

Interviewees

69

Bibliography

70

Closing Remarks

72

Research team73

3

01 DIGEST

What are socially inclusive
cultural projects?
In the first chapter, we deepen our understanding of how cultural projects
are evaluated and how such projects can be socially inclusive, and thenconsider how evaluation methods can connect these two questions.

Approaches to evaluation
࡛ Process evaluation

・Checking both the logic and the management that led to project results
・Focusing on the conditions that bring about empowerment and serendipity

Important points
regarding evaluation
࡛ Choosing a method that suits the goals of the project࡛ Measuring “value” from multiple perspectives
࡛ Staying aware of the differences between goal
and objectives

࡛ Outcome evaluation

・Paying attention to how outcome evaluation is different from output evaluation
・Articulating values through participatory evaluation
・Analysing in light of timeframes, spheres of influence, needs, and potential outcomes࡛ Project reports

・Using less biased styles of writing
・Understanding and illustrating changes before and after project implementation
・Recording the reasons for and significance of these changes in working toachieve the goal࡛ Advocacy

・Disseminating results effectively
・Communicating strategically

Socially inclusive
cultural projects
࡛ Social inclusion is the idea of building a society thatembraces diverse groups while respecting differences.
࡛ It is important to solve problems by creating new values.

4

Summary
Evaluation is a means, not a goal.

Evaluation is a site of communication.

Evaluation is an inherently creative act.

5

02 DIGEST

Before evaluation
What problems can come up when groups doing cultural projects
do evaluation? What do they need to watch out for?

1 Who conducts the evaluation?
࡛ Rather than simply bringing in an outside evaluator, socially inclusivearts projects should be evaluated by all involved.
࡛ Make evaluation participatory, collectively determining what kind ofevaluation is needed.

2 What to evaluate?
࡛ Discuss in detail the purpose of evaluation and the target audience.
࡛ Facilitators need to consider that the targets for an objective are asdiverse as the participants themselves.

4 When to evaluate?
࡛ Understand evaluation as a cycle and design evaluationthat involves gathering data and engaging in reflection fromgoal setting onwards.
࡛ The real joy of social inclusion arts activities is whenthings occur that could not have been imagined during
the planning process.

Summary
Customise evaluation methods to suit the site and goals.
Develop a shared awareness of what makes the projectvaluable—your collective sense of what is important inthis process.

As a group, figure out how to communicate this awarenessto other people. This process will help you develop a

3 How to evaluate?
࡛ Data gathering methods should be appropriate to what is

possible on site, being aware of the differences between“process evaluation” and “outcome evaluation”.
࡛ The importance of qualitative studies that do not impose anumerical framework are reassessed among experts.

6

shared vocabulary.

It is essential to have an opportunity
for participatory evaluation thatemphasizes mutuality and collectivity.

7

03 DIGEST

Evaluation wisdom in practice1Consciousness change in project participants

Organisations

This chapter offers findings from interviews with experts and projectgroups on evaluation in practice. Look at the examples that are closer
to your goals and your analysis of the issues to find ideas on how tomake evaluation work for you.

P42

Ryukyu Philharmonic Orchestra, general incorporated association4

tonge of people
ith a broad ra surveys.
w
k
or
w
to
t
I wan
and
tion indica to rs
create evalua

2

Deepening partnership

Organisations

Advocacy through evaluation

Organisations

Kitakyushu Performing Arts Center（ Kitakyushu City Foundation for Promoting Arts and Culture ）sulated over longer per iod
I want to use data accum
ent.
em
nag
ma
al
ion
zat
ani
in project reports on org

5

P44

Human resource development within organisations

Organisations

BEPPU PROJECT, a non-profit organisation( Secretariat of the “Mixed Bathing World” Executive Committee )I want evaluation to strengthen our group structuresand develop human resources.

8

Psychological approaches to developing indicators

Organisations

Tanpopo-No-Ye Foundation, Kinki Labour Bank

P50

Tokyo Bunka Kaikan( Tokyo Metropolitan Foundation for History and Culture )I want to measure the impacts ofmusic improvisation workshops
RQRXUROGHUSDUWLFLSDQWVLPSDFWVWKDWPD\EHGLIƓFXOWWRLGHQWLI\ns
with corporatioin partnership
ed
n?
at
ig
cre
al
ts
es
ec
lu
Arts projs and vare that our goal
How do we ensu

3

P48

P46

6

Medical approaches to developing indicators

Organisations

P52

Sendai Tomizawa Hospitaltherapyre the results ofes.
I want to measu
qu
ni
ch
te
a
am
of dr
that makes use

9

04 DIGEST

Communication through evaluation

How should we evaluate
cultural projects?

I wish arts organisations would
use quantitative indicators intheir self-assessments.
There are limits towhat questionnaires
can give you.

Government

The written reports helpedme explain the project tomy boss effectively.

Evaluation design

Indicator review

Data gathering

Result utilisation

It is essential to consider thegoals of evaluation before
thinking about evaluation
methods. Think of evaluation
goals as part of your projectplans.

It is possible to quantitatively
measure not just outputs,
but outcomes. Consider
what kinds of indicators will
best capture project results.

Gather data with a variety ofmethods. Archive records as
the project proceeds.

Design a strategy for theapplication of evaluation
results. Who will be theultimate recipients of yourproject results?

Artsorganisations

10

We envision evaluation as a hub for communication between projectstakeholders coming from a diversity of perspectives. We look at
key points on communication through evaluation in various situations,p o i n t s t h a t c a n a s s i s t i n i m p ro v i n g p ro j e c t m a n a g e m e n t a n ddevelopment.

Evaluation isn’t
something thatyou do after the
project has ended.

What exactly are
indicators?

Shouldn’t we have surveyed
the audience at the event?

It’s hard to convey thevalue of our activities.

11

What are socially inclusive
cultural projects?
When the topic of evaluating culturalprojects comes up, discussions tend
to quickly turn to evaluation methods
and indicators. But first, clarify why
something is being evaluated and thenfind the best way to fulfill that purpose.
In this chapter, we will consider what
is important in evaluation, deepen our
understandings of socially inclusive
cultural projects, and then find ways toconnect these two.

12

13

Evaluation for what?

The first step of the evaluation process is to identify the purposeof evaluation. If the evaluation method that does not suit the
purpose, the project may be led in the wrong direction.
A cultural project might be evaluated in order to:࢒ Critique

Describe the positive and negative aspects of the output, i.e.,
the works, events, etc.

࢒ Enhance processes

Check project processes to improve efficacy

࢒ Assess impact

Identify the social changes (outcomes) brought about by the
project

࢒ Report

Examine and report on project implementation

࢒ Monitor policy impacts

In reporting, the main purpose is to assess whether the project
has been implemented in accordance with the plan. Such
reporting would ideally be used as basic data for policy assessment,
but unfortunately this is not happening in Japan. In addition tousing project data for reporting, then, practitioners can takean advocacy approach to communicate the outcomes andsignificance of their activities to society, while administrative
officials must communicate their performance and potential tofinancial officers to appeal for funding. Advocacy focuses on how
to convey outcomes and impacts in broader terms.
As such, evaluation refers to a wide range of methods in relation
to an array of objectives. Methods such as the logic model and
SROI (social return on investments) are not adequate in all
circumstances. Identifying the objectives and choosing theright methods is the critical first step.

What are socially inclusive cultural projects?

What are socially inclusive cultural projects?

1

Measure the results of government programmes

࢒ Advocate[Practitioners]
Communicate the significance of the project to society[Administration] Leverage achievements to enhance potential
funding opportunities

Ah ha! We should select an evaluation
method according to our purpose!

Different objectives should be evaluated differently. For example,
evaluation to critique should assess an output (work, performance,
event, etc.) according to the criteria of the particular genre.
Evaluation to enhance a project involves checking projectprocesses. In evaluating impact assessment, the focus is on
outcomes (social changes brought about by the project) rather
than outputs.

14

15

Evaluation is not measurement

The literal translation of evaluation from the Japanese term is“valuing figures.” Although evaluation and measurement are often
considered identical, measurement can only provide figures. But
evaluation is much more than this.
Let's say that a glass has been filled about halfway with water. If
you measure the amount of water with a fine scale ruler, thenumber is a big one. But if you use a rough scale ruler instead, thenumber becomes smaller. Since differences in criteria havechanged the figure, it is not clear what has really been shown in
the figure.
Next, let's assume that water takes up half the capacity of theglass. How do we interpret this? As the adage goes, should theglass be described as half-full or half-empty? Evaluation is how
we assign value to the results of the measurement.
If the glass had never had so much water in it before, it could bereasonable to say the glass is half-full. But if the glass has always
held this much water or more, pointing out that the glass ishalf-full will not be of much consequence. What basis do we use
to make these decisions?

Even if we explain basis for a decision, if it’s only one person's
decision, others may argue that it’s simply one subjective
interpretation. So listening to the opinions of people from different
standpoints is important. Listen to people who see the glassfrom above or below, people who see it from a distance, andpeople who have seen many different glasses. More unbiased
judgments and nuanced conclusions are reached by gathering
many different perspectives.
In evaluation, it is important to express not only figures, butvalue. An integrated view from different perspectives provide abasis for unbiased judgment. It is in the spaces in between various
perspectives that evaluation exists, not in a single individual.

half-full!
Half-empty...

?

16

What are socially inclusive cultural projects?

What are socially inclusive cultural projects?

2

Which scale do we use?

17

Goals or objectives

The terms “goal” and “objective” are often used interchangeably,
but in project and evaluation design, there are important distinctions
between these terms. (Note that the direct translation of theJapanese word for goal is “target”, while objective is “signpost.”)The goal is the end that a project pursues. In the case of publicprojects, goals refer to changes in society that we want to achievethrough particular activities. Holding an event is an output, not an
outcome, so it is not considered a goal. A goal should articulate
what can be achieved over the medium- to long-term through theongoing implementation of similar projects or in collaboration
with others, rather than by one project alone. Some examples ofgoals are:

While a project's goal should remain unchanged, objectives may
shift and develop, adapting to the process to become more
appropriate and realistic.
Things never go quite as expected, least of all in cultural projects.
Nothing is gained by carrying out plans at the expense of theultimate goal. We must focus on the best things we can do today
and handle changes as they come up, keeping a close eye on ourgoal all the while.
Goal

Right

࢒ to create relationships that build empathy among people withdifferent values,

࢒ to build new communities of diverse people, orWhat are socially inclusive cultural projects?

What are socially inclusive cultural projects?

3

Left

ay

Ropew

Sea

࢒ to propose models in which diverse people collaborate.
An objective, on the other hand, is a target a project works
toward in order to achieve a goal. The phrase “numericaltarget” is often used, but the objective does not have to bequantifiable. Objectives could be:࢒ to create opportunities for people with different values to interact,
࢒ to deepen understandings of disability in society, andEach route has
its positives.

࢒ to create a new sense of shared values in the community.
Objectives differ from goals in that they can be measured bydegree. In addition, although there is usually only one goal, it’scommon for a project to have multiple objectives.

18

Which objectives will lead us to our goal?

19

What are socially inclusive cultural projects?

Social inclusion is the idea of building a society that embracessocially marginalised people. (See our first handbook, “Social
Inclusion Through Culture and the Arts”, for details.) When social
inclusion is promoted as a cultural project rather than a social
welfare initiative, there are two concepts coming together:
problem solving and value creation.
First, the problem to be solved is the marginalisation of various
groups in society. The social positions of people in “minority”
groups have been determined against their wills, and their ability
to behave as they like is restricted, which makes their lives
difficult. Marginalisation of specific groups in society is theresult of systemic barriers. Arts activities in which dominantideas of right and wrong are challenged can provide an
opportunity for marginalised people to express themselves
freely, transgressing the social boundaries imposed on their
behaviour. In everyday life, for example, “doing the same thingover and over” or “doing nothing” is rarely appreciated, but in arts
activities, these behaviours are sometimes used as a means ofexpression, thereby empowering the participants. This is an
example of problem-solving through arts activities.
The concept of value creation involves serendipity: the unintentional discovery of something valuable. Suppose two people ofdifferent backgrounds are creating together. If one person’s ideaof what is “good” is different from the other’s, it might be difficult
to produce work, but when they come upon something that theyboth think is good through a mutually respectful process, a workof art can take shape. This is serendipity. As the work takes

20

shape, the collaborators form a new sense of what is valuable.
As such, arts activities can produce artworks and at the sametime develop systems for appreciating these works in social
context. This is the creation of value through arts activities.
It may seem difficult to achieve a balance between solving problems and creating value, but in practice, creating value leads to
the resolution of problems. Experienced artists and coordinators
can tell what sort of ideas will likely solve problems. This gut
feeling will be acquired to some extent, if you learn the idea of“process evaluation” methods, which are described in the next
section.

What are socially inclusive cultural projects?

What are socially inclusive cultural projects?

4

Problem-solving through value creation
I want to eat sushi but
I m not into raw fish. Avocado

It s not sushi
without f ish!

imitation crab meat
nori

There isn t much
fresh f ish
in California.

Ta-da!
!
California rolls
Now we caneat sushi!

This is pretty
tasty.

21

Process evaluation

There are two sides to process evaluation: theory evaluationand management evaluation. Theory evaluation checks whether
the project’s content suits its goal, and management evaluation
checks for any operational problems. Evaluators can tell whether
the project is likely to lead to social inclusion by checking against
the three points below, when the project plans are set, when
the project enters a new stage, or when the project requires achange.

Theory evaluation

Are the goalsconsistent withthe content?

Management evaluation

Were there any
problems withthe management?

First, regarding problem-solving (particularly when related toempowerment), it is essential to ① ensure a safe and secureenvironment, and ② develop a climate in which people arefree to express themselves at their own pace. Rather than
forcing people into things or pushing values onto them, facilitationmust affirm a diversity of expression while drawing out eachindividual's unique strengths. This is even more empowering if③ one person’s expression can intermingle with others’ tocreate new work. When people sense that they have contributed
to something, they may feel happy to be alive, or that they have aplace where they belong.

22

Next, regarding value creation, or serendipity, the most important
thing is ① play. This is play in the sense of allowing for a gap or amargin rather than planning everything perfectly, and play in thesense of incorporating playful elements. The next most importantis that ② everyone is equal, whether or not they are artists. As
these activities create value systems that have never existed
before, even the artist is not necessarily more skilled than anyoneelse. Another important element is ③ dialogue, which may
involve non-linguistic communication. It is important that peopleinteract directly with each other. In other words, value creation
requires situations in which there is space in the plan. The
presence of play enables dialogue within mutually respectful
relationships and leads to the development of work that engages
all participants.

What are socially inclusive cultural projects?

What are socially inclusive cultural projects?

5

To sum up process evaluation:① Does the space ensure a safe and secure environment thatincorporates elements of play?
② Do participants have freedom of expression and freedom to go
at their own pace, in egalitarian relationships with each other?
③ Are both linguistic and non-linguistic dialogue valued?
Are there situations in which the expressions of various
people connect to create artwork?

23

Outcome evaluation

Outcome evaluation examines the changes to society as a resultof the project. While process evaluation examines planning andmanagement in implementation, outcome evaluation considers
change more broadly. Outcome evaluation is often confusedwith the evaluation of outputs, such as event content andnumber of participants.
Socially inclusive cultural projects must identify key outcomes inlight of project goals from two perspectives:① how they contributed to solving social problems, and② what new values were created.
Participatory evaluation, in which a range of stakeholders
come together to engage in dialogue, is effective in identifying
outcomes (see Chapters 2 to 4). As dialogue continues, thegroup begins to find the words for what at first was difficult to
explain. It is easier to organise outcomes along two axes:
timing and scope, as shown in the table below. As topics
emerge, dialogue will ramp up, and more essential outcomes will
emerge.
Select outcomes important to the project goal, and the ones
whose causal relationships are easily explained. Different
approaches lead to different outcomes, even among culturalprojects with the same goal, so it is important to consider themcarefully. The creation of new values is a key aspect of culturalprojects, so a clear articulation of the relationships between
goals, outcomes, approach, and values is important.

24

Immediate

Medium-term(changes have(changes havealready happened) begun to appear)

Long-term(future changes,
logically projected)

Individual participants(behaviour, consciousness, attitude, understanding,interest, skills, living conditions, etc.)

Stakeholders involved in the project(organisational, operational, relationships, etc.)

Local communities and society as a whole(consciousness, interest, social mechanisms, social
situations, etc.)

What are socially inclusive cultural projects?

What are socially inclusive cultural projects?

6

Some of these changes may become indicators. One might thinkthat the term “indicators” refers to standards that already exist,
but they can also be created. Once an outcome has been identified as important, some of the elements needed to achieve thatoutcome can be selected and indicators that specify how often
and to what extent those elements are present can be created(see Chapter 3). Once such indicators are defined, they can beused to verify certain benefits of future projects.
Outcome assessment is likely to be conducted after the completion of a project. When planning projects, however, keep in mind
that data must be collected throughout implementation.

25

Project reporting

When a project is completed, the implementing organisation
usually submits a report to administrators and funders. Consider
the following points when preparing the report.

1. Describe the changes between pre-implementation (before)and post-implementation (after).
2. Write in a manner that makes it easy to understand who isevaluating the project, on what basis, and from whatperspective.
3. When describing a numerical value, explain how it wasderived and what it shows.
4. Include opinions from diverse perspectives and stakeholders.
5. Make comparisons in simple terms, like “compared to lastyear” or “compared to similar projects.”
6. Describe and discuss the positives as well as the areas thatneed improvement.
7. Explain the need for changes to objectives and projectcontent in order to achieve the goal.
8. If there were unforeseen developments during implementation,
describe the process of such developments in detail andpromote these as additional or alternate project outcomes.
9. ･･･
10. ･･･
11. ･･･

26

8

Advocacy

We tend to assume that others have the same ideas as us about
what is important or good. But no matter how much the culturalvalue of a work is promoted, if a person places more importanceon economic value, the conversation can’t go far. At the sametime, it would be absurd to emphasize economic ripple effects
when discussing a socially inclusive cultural project. As a result,
it is essential to communicate the significance of social inclusion arts activities to funders and to society as a whole in asustained way. This is advocacy.
Advocacy is often conducted through publications, videos,
websites, and publicity, but strategic communication in everyday
life is also important. For example, administrative personnel
changes every few years, so it is necessary to explain thesignificance of a project to new officials with each change.
Cultural organisations need to work together to raise theawareness of legislators and administrative officials at every
opportunity. In addition, it is essential to actively and regularly
communicate with people from different industries and fields,
calling for cooperation and support across social boundaries.

What are socially inclusive cultural projects?

What are socially inclusive cultural projects?

7

Today's neoliberal society places utmost importance on economic
indicators in policymaking. Life has become increasingly difficult
as economic value becomes the key measure of people's
existence and behaviours. Socially inclusive projects are among
the most effective approaches to nurturing a multifaceted senseof value in society. We need to work together in a consistent way
to convey the significance of these activities to all levels ofsociety.

27

Summary
Eva luati on is a me ans rat he r th an an end .
Evaluation is not conducted for evaluation’s
sake, but to maximize project results. It isessential to clarify the goal and choose methods
appropriate to it.
Evaluation is also a place for communication.
Verifying the logic and processes that lead toproject outcomes, determining actual outcomes,
and bringing attention to the significance of the
project through advocacy—all of these could be
described as opportunities for communication.
Finally, evaluation is an inherently creative act.
As described in the introduction, evaluationmeans “to find the value of”. To find the value ofsomething, we must be creative, which typicallymanifests in the dialogues in p a r t i c i p a t o r yevaluation.
Project evaluation should be conducted alongwith the project, rather than as an afterthoughttacked onto the end of a project. Perhaps it
should be regarded as a necessary effort that asocially inclusive project takes to achieve its
goal.

28

Before evaluation
There are countless reasons for andmethods of evaluation. Tailoring
evaluation methods to a particularsetting with an awareness of common
methods and concepts is a good way
to begin. In this chapter, we explore
the difficulties that can arise whenorganisations conduct evaluationsof cultural projects they have
implemented, and what to consider
in dealing with such challenges.

29

Who evaluates?

only to do activities
It is important not.
but to evaluate themso you evaluate!
You re the expert,
I hope you do well!
I ll leave it to you.

The evaluation of social inclusion arts activities is not
something that can be done by an outsider alone. It must be
something that everyone creates together through participatory
evaluation. It’s vital to hold a consultation process in whichstakeholder opinions are sought out, discussions are held
regarding project aims, and appropriate evaluation methods
are determined.

Before evaluation

Before evaluation

1

KEY WORD

What?!

You can t just leave it to me …

The idea of evaluation generally evokes an image of experts witha wealth of specialised knowledge. Since social inclusion arts
activities are conducted in art and welfare fields, a wide
range of knowledge is required for evaluation. But few
specialists have knowledge of both art and welfare. As a result,
there is often a disconnect between the kinds of evaluations thatorganisations really want and the kinds of evaluations they can
get when outsourcing to an expert.

30

Participatory evaluation
Participatory evaluation includes all stakeholders in theevaluation process, including participants, facilitators andothers involved in implementation. Such a process can reveal
the significance of the project in a multifaceted manner andraise collective awareness. It involves considering the opinionsof stakeholders at the design stage of the evaluation, anddiscussing the data gathered and analysed.
Bibliography

P. 70 ⑧31

What to evaluate?

I want participants to really feel alive.

But some participants may have other goals.
It's okay for each of us to have different
goals, isn't it?

When setting the goal for evaluation, carefully consider theaudience for the evaluation and stay aware of the difference
between an aim and a target, as discussed in the previous
chapter. When setting a goal, facilitators need to be awarethat the targets for achieving the goal are as diverse as theparticipants themselves.

Before evaluation

Before evaluation

2

KEY WORD

Facilitation
That's right. In a single activity,

there are diverse people withdiverse thoughts.

On the previous page, we discussed how people involved in
activities take part in evaluations in various ways. At the actual
evaluation site, though, you might struggle to articulate theimportance of your activities to your peers, and the significanceof evaluation might be lost on people. If you don't build theevaluation process in a way that feels right, you'll lose sightof why you are doing it and develop evaluation fatigue: feelingas though you’re evaluating for evaluation’s sake.

32

Facilitation means to help forward. It refers to the ways you
can help a group of people engage in an activity, such as
solving a problem or coming up with new ideas. Facilitatorsplay an important role in a creative process by taking aflexible approach and encouraging discussions in which
diverse participants find common ground.

Bibliography

P. 70 ⑦⑩

33

How to evaluate?

I studied social impact assessments
in a recent seminar.
I'd like to try them for our activities.
I'm going to have an event the day after tomorrow.
What kind of questionnaire should
I write to evaluate the project s social impact?

It is important to experiment, creating indicators and collecting
data through trial and error according to what is possible in situ
(see Chapter 4). Arts evaluations do not boil down simply toparticipant numbers or attendance rates. Various qualitative
studies have also been carried out to engage in evaluation withmeasurements other than numerical values.

Before evaluation

Before evaluation

3

work sheet

KEY WORD

What?! I don t think social impact
assessments work that way...

Various terms have emerged to describe evaluation methods andapproaches – Social Impact Assessment, Programme Assessment, Logic Models, SROI. Some people seem to believe thatthey have to follow these methods. It is important that methods
be designed to suit the evaluation goal. What can be evaluatedand how it should be evaluated depend on the situation. There isno universal method.

34

Qualitative study methods
Unlike quantitative surveys, which provide somewhat“scientific” knowledge, qualitative surveys collect data on
specific situations for particular problems, and interpret
results in a limited way. This could include interviews,
fieldwork, and observations by advisors. Some qualitative
surveys awaken the reader’s sympathy just as a great
anecdote does. Combining different methods of quantitativeand qualitative investigation may reveal truths that havegenerally been obscured.
Bibliography

P. 70 ①35

When to evaluate?

We know just surveying the audience atthe event is not enough.
We can learn a lot from what the various
people involved were thinking throughout
the project.

Think of evaluation as a cycle and design a unified structure,from goal setting to data collection and review. The real thrillof socially inclusive arts activities is often what happens
unexpectedly; things that could not have been imagined
during evaluation design. Record these serendipitous events
thoroughly and capture them in the final evaluation to refer to inyour next project’s evaluation and planning.

Before evaluation

Before evaluation

4

ONE WORD

Field notes
That's right.
Anecdotes, which can t be expressedin numbers, can illustrate so muchabout a project.

As mentioned in the Outcome Assessment section of the previous
chapter, when an evaluation begins after the activity has been
completed, it should communicate a sound understanding of
the nature of the activity.

What is the best way to record when something unexpected
happens? It can be useful to write field notes, as is donein cultural anthropology or sociology. This will allow the
day-to-day circumstances of activities to be viewed fromboth a bird's eye view and in close-up. Such notes canclearly describe how the people involved and the observers
themselves develop their awareness of and engagement in
the activities. Just like field notebooks, these methods can insome cases be used to keep detailed records of workshops
and so forth.
Bibliography

P. 70 ②

It can be better to gather data beforehand, or to gather data
afterward. It is important to comprehensively design methods
and timelines in advance, based on the evaluation goal.

36

37

Summary
There is no one w ay t o e valu ate the field ofsocially inclusive arts activities, because it's acase-by-case basis for what constitutes a value
criterion and what can be used to persuade
government, funders and others. Communicate thevalue standards applied and persuade funders andadministrative agencies of the impact of theactivities. Evaluation methods that might be usefulin other contexts will not necessarily suit yours,
and that disconnect can lead to evaluation fatigue.
Evaluations may be conducted to promote thesignificance of the activities, or simply to ensurethat activities are implemented and effectively
connected to next steps. Tailoring evaluation
methods according to the particular site andgoal is essential to success. To do so, the people
involved need to develop a common understandingof what is important and why this has value.
Developing new language and terminology throughdiscussions with diverse stakeholders is also a key
part of the process. How will ideas or values thathave become common sense to participants beconveyed to others?
Ideally, evaluation is never unilateral; creating
opportunities for communication among diverse
stakeholders is essential.

38

Evaluation wisdom
in practiceThis chapter summarises some key ideas on
evaluation through the words of evaluation
experts working in various fields andorganisations. Although not all the examples
are related to social inclusion, they areo n e s in which various participants andstakeholders are involved in evaluation. By
reading these examples that are similar toyour work in terms of project goals, issues,and so on, we hope you can explore ideas
about how to plan and implement futureevaluation activities more effectively.

39࡛ approaching a broader range of project stakeholders 1 2 ,࡛ developing human resources within organisations in connection
with advocacy 3 4 , or
࡛ creating benchmarks unique to the project 5 6 .

3Human resource development within organisations

P46

Hiroki Miura

Advisor, Oita Prefecture Arts, Culture and Sports Promotion
Foundation; Director of Research Department, Oita Associationof Corporate Executives

Since 2014, Hiroki Miura has worked at the Oita Prefecture
Arts, Culture and Sports Promotion Foundation, engaged
in research on how to implement and evaluate cultural
policies and funding mechanisms for local art projects.
He is certified by the Japan Evaluation Society.

4

Advocacy through evaluation

P48

Evaluation wisdom in practiceEvaluation wisdom in practice

Tips for practice from the six examples
6
This chapter lists key points of evaluation in Q&A format, which havebeen gained from interviews with evaluation experts. Although not
all of the examples take a social inclusion approach, they are examples of either:

Mitsuhiro Yoshimoto

Director, Center for Arts and Culture, NLI Research Institute

Please refer to the examples that are closest to your style of activities
and to the goal of your evaluation.

1

Consciousness change in project participants

P42

Chika Ochiai

Mitsuhiro Yoshimoto is engaged in research in a wide
range of arts management fields, including cultural
policy, public theatre and hall management, creative
cities, the Cultural Olympiad, Arts NPOs, arts in
education, and private support for the arts. In addition,
he has been involved in developing theatres, concert
halls and art galleries as a consultant.

5

Nahoko Kusaka

Director and Chief Knowledge Officer, K-three Inc.; Researcher,
Graduate School of Media and Governance, Keio University

3URIHVVRU)DFXOW\RIb&RQWHPSRUDU\6RFLDO6WXGLHV'RVKLVKD:RPHQ Vb&ROOHJHRI/LEHUDO$UWV

Chika Ochiai has been working as an evaluation
expert in organizational and project management,social outcome investigations, children’s support
through the arts, community revitalisation and social
inclusion projects.

2

Deepening partnership

P44

Taro Okabe

40

Psychological approaches to developing indicators P50

Specialising in Clinical and Geriatric Psychology, Naoko
Kusaka is focused on research and practice in programs
working towards the meaningful lives for the elderly.

6

Medical approaches to developing indicators

P52

Masahiko Fujii

Managing Director, Tanpopo-No-Ye Foundation

Clinical Professor, School of Medicine, Tohoku University

As a member of the Tanpopo-no-ye staff team since2003, Taro Okabe has been working on exhibition
planning and production, public relations, and directing
workshops and seminars.

President of Sendai Tomizawa Hospital and of the
Yamagata Health Hospital, Masahiko Fujii’s researchareas include elderly medicine, dementia, and thedevelopment of nursing care equipment.

41

Consciousness change in project participants

Organisations

Ryukyu Philharmonic Orchestra, general incorporated association

Project Name

Yuimaru Music Project “Beauty Sounds Concert”

Project Overview

Concerts were planned and held with experts in various fields suchas music and welfare, musicians with disabilities, and families of
persons with disabilities. The concerts were enjoyed by persons withdisabilities and their families. (2019)

Let’s ask
Ms. Chika Ochiai!

People involved in evaluation design
Chika Ochiai（K-three Inc.）・Ryukyu Philharmonic Orchestra
・Musicians with disabilities, families with disabilities,
supporters
・University professor, Council of Social Welfare, Social
welfare company
・Administrative staff, prefectural board of education,
prefectural culture promotion association

Other participants
・Visitors
・Persons with disabilities andtheir families/supporters

Project members

People involved in business operations
Orchestra members, Volunteers

Q. What did you do to evaluate this project?
A. We conducted a kind of participatory evaluation in order to use the evalua-tion process to improve future projects. In particular, we ① clarified the goalsof the project and verified what changes had actually occurred, and ②

listened to participants' opinions directly, clarifying outcomes and issues
with the project.

Q. Please tell me how you facilitated the evaluation.
A. First, we gathered opinions by asking six questions, including: What kind of

results do you want project members to achieve in one year? What kind of team
do you want to become? What do you want to learn through the survey?
Next, based on the opinions gathered, a simple logic model was constructed to
articulate the results that we would aim to achieve. Indicators linked to theoutcomes were prepared, and the contents were checked and examined byproject members to decide which of the desired outcomes were mostimportant and which ones we wanted to measure.
On the day of the event, I acted as a volunteer staff member and spoke directly[V [OL Z[HMM TLTILYZ HUK JVUJLY[° H\KPLUJL [OH[ 0 KVU»[ \Z\HSS` ZLL HUKinterviewed them indirectly. Even after the concert, I called myself a staff
member and interviewed them this way.
After the concert, we analysed and summarised the results of questionnaires
and interviews, and shared the results with members to discuss what they could
improve for next time.

42≫Stakeholders

Evaluation wisdom in practiceEvaluation wisdom in practice1

I believe that evaluation is a powerful tool that can put forward theopinions of each person involved in the project.

Q. How did the process affect the people involved?
A. Over the past several years that I have been conducting evaluations for thisproject improvement, it has become habitual for the organisation to evaluate.
;OLYLPZHNYV^PUNZLUZL[OH[[OLVYNHUPZH[PVU°^HU[Z[V\UKLYZ[HUKHUKSLHYU
from the opinions of participants in order to improve projects. As the significance of the activities are shared with a broader range of stakeholders,
unintended changes have also emerged, such as giving rise to newpartnerships and connections, and increasing stakeholders’ motivation.

Q. What do you keep in mind when creating questionnaires and reports?
A. When preparing materials, always be aware of who you are writing them for.

For example, if a questionnaire is for children, we are careful to simplify thewording to make it more readable. For our next report, we have held discussions with group members and put a questionnaire on our website withannotations that make it easier to read aloud than Japanese normally is, sothat people with disabilities can understand the text more easily. I think it isimportant when creating reports that you summarise them, always keeping in
mind who will read the report and how they will use the information.

Bibliography

P. 71 ㉓㉘

43

Deepening partnership

Organisations

Tanpopo-No-Ye Foundation, Kinki Labour Bank

Project Name

“Human, Art, Community” Project

Project Overview

Implemented in different regions of the two Kinki prefectures every year, with
the Kinki Labour Bank organising and Tanpopo-no-ye in charge of planningand operation. Conducted research on local art- and welfare-related NPOs,
companies, shopping districts, key persons, etc., and held exhibitions,workshops, forums, and so on tailored to the region. (2000-2019)

Let’s ask
Mr. Taro Okabe!

Q. What did you do to evaluate this project?
A. We focused on listening to the opinions of each person involved and listening

to participants with the aim of capturing changes that occur in local communities through art. Needless to say, we also conducted questionnaires forvisitors. As each person's impressions are complicated and diverse, I think
it is important to pay attention to the complexities of their responses
without simplifying.

Q. Is there anything you keep in mind when reporting andA.

sharing communications with relevant parties?

There are times, for example in structured meetings such as project briefing
sessions, when people are hesitant to express their opinions due to social
hierarchies.
Since project completion, we have been setting up forums such as
Review-and-Drinks Parties, where stakeholders get together and work onreports. Watching slideshows of the project in an informal setting, people
share their true feelings: “Oh, I remember that!” “That was really fun.” Thatway, positive opinions surface and clarify what the group wants to do next andhow you can make it happen.

≫Stakeholders
People involved in evaluation design

Other participants
Visitors

・Taro Okabe (Tanpopo-no-ye General Foundation)
・Kinki Labour Bank・Arts/Welfare NPO Group・Social welfare corporation
・Local residents

People involved in business operations

Q. What kind of relationships have you built with the corporate partner ofthe project?

A. From the start, the philosophy behind our activities had a lot in common withthe corporate philosophy of Kinki Labour Bank. Carrying out our project has
meant getting involved not just with clients and trustees, but also working
together to build relationships that enable us to cooperate by making the mostof our networks and expertise. We have been able to continue this long-standing relationship, which has made it possible, for example, for employees at
Kinki Labour Bank's initial orientation training to visit our project site and learn
about why Kinki Labour Bank's commitment to this project matters from an
NPO point of view.

Q. Sometimes corporate partners are overly concerned with participant

QXPEHUV+RZGR\RXJHWWKHPWRSD\DWWHQWLRQWRDSURMHFW·VFRQWHQW"

A. If you have a partner who judges whether a project is good or bad based only
on participant numbers, you will only ever end up with the result, “We didn't
have enough people this year.” We had good discussions with Kinki Labour
Bank about attracting clients by segmenting them based on different public
relations issues and focusing on what kinds of changes have occurred in our
project activities. Even if the number of participants is small, you can still havemeaningful discussions by digging into specifics, such as the kinds of people
who got involved through this opportunity, and the kinds of changes that
occurred in the region.

Bibliography

44

Evaluation wisdom in practiceEvaluation wisdom in practice2

P. 71 ㉔45

Human resource development within organisations

People involved in evaluation design

Other participants

Organisations

BEPPU PROJECT, a non-profit organisation
:LJYL[HYPH[VM[OL¸4P_LK°)H[OPUN°>VYSK¹°,_LJ\[P]L°*VTTP[[LL

Hiroki Miura（Oita Prefecture Arts, Culture and Sports
Promotion Foundation）

Visitors

Project Name

Beppu Contemporary Art Festival 2015 “Mixed Bathing World”

Project Overview

Activities aimed at promoting arts and culture in Beppu City, contributing
to the development of personnel who will lead the revitalisation of thelocal community, and promoting the city throughout the country.
The “Beppu Art Month” civic cultural festival is held annually, as wellas the art festival “in BEPPU”, which takes the form of a solo exhibition.
(Beppu Contemporary Art Festival “Mixed Bathing World”, the predecessor to in BEPPU, started in 2009.)

・BEPPU PROJECT STAFF
・“Mixed Bathing World” Executive Committee

・“Beppu Art Month”
Registration group

Let’s ask
Mr. Hiroki Miura!

Q. How did you evaluate this project?
A. We undertook participatory evaluations (developmental evaluations) with the aimof ① strengthening the foundation of the organisation including developing
human resources, and ② ensuring accountability.

Q. What specifically did you do?
A. We introduced the Balanced Scorecard (BSC)* concept. Representatives of theorganisation drafted a set of strategic objectives based on five perspectives:
regional revitalization, audience, stakeholders, finance, and management. But
on-site staff told us that they didn't get a gut feeling that the strategic objectives
would be right, and that they didn't have a concrete sense of what they were. We
believe that it is important to examine our strategies based on our staff's
sense of the context, to deepen their understanding of evaluation, and to getthem fully on board.
A total of five BSC study workshops were held between July and September
2017 with the participation of dozens of staff members.

*The Balanced Scorecard, popularised in the 1990’s, is a systematic way for organisations to track strategy
implementation and operations.

46≫Stakeholders

People involved in business operations

Q. How did you proceed with the workshops?
A. At the first one, we went over what BSC is, the current strategy of the entire

Evaluation wisdom in practiceEvaluation wisdom in practice3

project, and how we would evaluate it. We considered proposals for indicators
one by one, with an emphasis on creating useful indicators for ourselves, so that
evaluation would be something that guides us rather than something that we resent
doing. We began by considering and sharing our philosophical goals, andgradually proceeded to discussions on our work and management.

Q. What changes have you observed in the staff team?
A. In the day-to-day work, I heard a staff member say, “I just realised I achieveda part of an objective that everyone created together!” I felt like this was anopportunity to connect our own work to the project’s overall philosophy
and goal.

Q. What advice do you have for organisations that want evaluation to leadto staff development and awareness raising?

A. The first step is to discuss the goal of the project together and the path toachieving it. It can be a burden for staff to start a new survey, and an organisation
may not be able to afford to hire evaluation experts to do one. So begin byorganizing the data that you are already collecting. Based on that, I think you cancreate the simplest possible of logic models and indicators, and you can alsomeasure outcomes through questionnaires.

Bibliography

P. 71 ㉑47

Advocacy through evaluation

Organisations

Kitakyushu Performing Arts Center（Kitakyushu City Foundation for Promoting Arts and Culture）

Project Name

Annual Evaluations of the Kitakyushu Performing Arts Center

Project Overview

The Center, consisting of a large hall, a medium-sized theatre, and asmall theatre, engages in producing, staging, training and education in
performing arts. It has been conducting annual project evaluation since
its inauguration in 2003.

Let’s ask
Mr. Mitsuhiro Yoshimoto!

Q. What did you do to evaluate this project?
A. Since 2003, we have been implementing this evaluation on an ongoing basis tounderstand the outcomes, impacts and challenges with the programs andoperation of the Kitakyushu Performing Arts Centre. We are interested in
improving strategy to run the theatre better. In addition to the following fourstudies, we also conduct a study on a particular theme each year, allowing
us to organise study results based on the framework for the evaluation.

① Overview and outputs of each ﬁscal year
(collection and analysis of basic data on theatre management)② Characteristics of the audience and evaluation from the audience perspective(analysis of questionnaires administered to audience members)③ Evaluation from rental facility user perspectives(analysis of questionnaires administered to rental facility users)④ Analysis of economic ripple eﬀects and publicity eﬀects

Q. How are evaluations used in theatres?
A. In 2015, we held a briefing on the results of a 12-year evaluation survey
targeting all theatre staff, which we had used to review our operations. We
reported the changes in the performance of KPAC’s operations since
opening and the level of user satisfaction, as well as what makes the Kitakyushu Performing Arts Center special. I think that staff members are usuallybusy with their day-to-day duties, and they might not have the time to readan evaluation report carefully. Some of the staff members who participated in
the briefing session told me that they developed a better understanding of
the importance of the annual evaluation survey. Evaluation survey results
give you an opportunity to take a bird’s eye view of your work, and theyalso help you to empower staff.

48≫Stakeholders
People involved in evaluation design

Other participants

Mitsuhiro Yoshimoto（NLI Research Institute）
・Kitakyushu Performing Arts Center
・Directors, managers and staff members of KPAC

・Audience・People involved in the
programs and operation

People involved in business operations

Q. How can reports be better utilised?
A. I would like our evaluation studies to be used more actively, rather than just for

Evaluation wisdom in practiceEvaluation wisdom in practice4

the purposes of understanding outcomes and challenges or carrying out
plan-do-check-act cycles. It’s a shame when results and reports are only being
used in the theatre and its supervising governmental organisation. I believe it’s
important to use the results of our evaluation to convey the significance of
cultural institutions and projects, and to advocate for them by building
public and government understanding and support.

Q. How can organisations get started with advocacy?
A. It is important to visually and sensitively convey the information obtained from

evaluation and the project outcomes using illustrations, infographics, andimages of the project. For example, Arts Council England made a 5-minutemovie summarizing the outcomes and impacts generated by their funding,including interviews with people involved. But sometimes it’s not easy to do inthat way. In these cases, our method involves focusing on the outcomes of thepriority projects for the current fiscal year, setting scenarios and hypotheses inadvance, and conducting studies limited to data collection and interviews that
support these scenarios and hypotheses. It’s important to conduct evaluation
surveys not only for evaluation, but to publicise the social significance of cultural
institutions and the arts.
I believe that this approach will lead a greater understanding not only of thecultural institutions we run, but also of the necessity of art and culture, which inturn will help to form a broad base of support for cultural institutions and culturaladministration.

Bibliography

P. 71 ㉒㉕㉗

49

≫Stakeholders

Psychological approaches to developing indicators

People involved in evaluation design

Other participants
Elderly care facility users

Organisations

Tokyo Bunka Kaikan(Tokyo Metropolitan Foundation for History and Culture)

Nahoko Kusaka（'RVKLVKD:RPHQ Vb&ROOHJHRI/LEHUDO$UWV,QVSHFWLRQWHDP）

Project Name

“Sound Sandpit” music workshop method

Project Overview

As part of social inclusion efforts, musical workshops that emphasise“creativity”, “collaboration”, and “participation” are held at facilities forthe elderly and social welfare in Tokyo. (2018)

・Tokyo Cultural Center workshop leader
・Personnel of facilities for senior citizens・Musician (Music workshop supervisor / Trainer)

People involved in business operations

Q. What should a project do if it is difficult to put targets into words orLet’s ask
Professor Nahoko Kusaka!

indices?

A. I don’t start out by deciding on uniform standards of what makes something
Q. What did you do to evaluate this project?
A. This project was evaluated with the goal of creating programs that encourage
voluntary and creative activities among the elderly.

Since I am a psychology researcher, I listen to the organisers’ intentions for andparticipants’ expectations of the programme, and I use psychological theory toindex these intentions and expectations and to visualise them with numerical values and words. In this case, I carried out structural evaluations
through the observation of actual conditions, dividing the evaluation into
three layers, and measuring the following items.
Quantitative evaluation ・Measure vital signs such as heart rate to measure cognitive function andhealth・Measure personal characteristics such as social activities and personality

Program-specific・Take videos for records from multiple directions
quantitative assessment ・Observe voluntary behaviours of moving an arm or tilting the body to hold amusical instrument
・Measure gaze movements looking at each other's behaviour in order tomove cooperatively

Qualitative evaluation

・Analyse changes in participants’ behaviour as measured by second-tierindices in relation to interactions between participants and leaders’efforts
・Determine the growth of the group as a whole from changes in the role ofimprovised performers

1
2
3

good. Instead, I take an analytical style, finding standards of what makes
something good by carefully describing what actually happened.
This time, one participant said, “It would be nice if the situation would be likeplaying in a sandbox, using musical instruments.” So we observed the workshop
by paying attention to how people engage in a kind of “sandbox play”, where yourbucket and spade have been replaced with musical instruments. If people“became absorbed” in playing in the sandbox, we talked to them about whatkind of behaviour we could measure as “becoming absorbed” psychologically,
and then dropped it into an index that we could measure (the “second layer” on
the previous page). This way, I try to find answers with other people, as amember of the team; answers that I wouldn’t see on my own.

Q. What is the goal of your evaluation?
A. Through the evaluation of music workshops, professional artists can clarify

why socially inclusive cultural projects matter; why they are important.
Looking at figures can’t tell you why participants’ “voluntary actions” became
more active in this workshop. Focusing on a three-layered analysis shows
you that the effective accompaniment of professional musicians underpinned
participant actions, often blending sounds that were initially disjointed into
beautiful harmony. Understanding the role of an artist through these interactions allows the project to explain the unique value of the artist’s role in thecontext of your specific context and activities.

Bibliography

50

Evaluation wisdom in practiceEvaluation wisdom in practice5

P. 71 ㉖

51

Medical approaches to developing indicators

Organisations

Sendai Tomizawa Hospital

Project Name

Theatrical emotional therapy

Project Overview

One of a variety of emotional therapies for people with dementia.
Theatre experts work on reading a scene from a play and stimulating
emotions for approximately ten elderly people with dementia. (2014)≫Stakeholders
People involved in evaluation design
Association of Japanese Theatre Companies
・Dr. Masahiko Fujii (Clinical Professor, Tohoku University
School of Medicine)・Sendai Tomizawa Hospital・Japanese Association of Dramatic Emotion Therapy (NPO)

Other participants
People with dementia

People involved in business operations
Yusaku Maeda (Director, Actor)

Let’s ask
Dr. Masahiko Fujii!

Q. What is theatrical emotional therapy?
A. Since 2014, I have been working with Sendai Tomizawa Hospital on theatricalemotional therapy along with Hidetada Sasaki (Professor Emeritus, School of
Medicine, Tohoku University) and Yusaku Maeda, a performer and actor. In thistherapy, theatre professionals do readings of dramatic scenes in a group of about
ten people, stimulating the emotions of people with dementia. It has been foundthat dementia reduces the function of the neocortex, which is responsible forreasoning, while the function of the limbic system, which is responsible for emotion,
is not significantly impaired. In light of this, we have used a variety of activities thatare likely to affect the limbic system. As a result, emotional therapy using theatre
was found to be effective for people with limited cognitive functions.

Q. What kind of evaluation did you conduct?
A. We compared theatrical emotional therapy to a control group treated withconventional medication. Cognitive function tests (MMSE) and emotional
function tests (MESE) were measured in the two groups, and we found thattheatrical emotional therapy significantly affected emotional function, although
there was no change in cognitive function. In order to further analyse emotional
functions, we have created a new index called the Delightful Emotional Index.

Q. What is the Delightful Emotional Index?
A. The Delightful Emotional Index measures the degree and frequency of actions

such as returning greetings, saying “thank you,” and paying attention to others.
In dementia, it is essential to activate emotional function. Increasing this index
reduces distressing emotions, such as physical and verbal aggression, andimproves the behavioral and psychological symptoms of dementia (BPSD).
How well the person responds to the caregiver's efforts not only demonstrates
well-being, it has a positive effect on the caregiver's attitude. If caregivers arethen able to respond affirmatively, a positive cycle is created, increasing thesatisfaction of people with dementia.
>LHYLJ\YYLU[S`JVSSHIVYH[PUN^P[O°(ZZVJPH[PVUVM1HWHULZL;OLH[YL*VTWHnies,
which is using these indicators to support advocacy efforts by compiling estimatesof reductions in drug usage and medical costs as a result of these activities.

Q. What is important in this effort?
A. The performer, Maeda, says, “Directing and acting don’t resonate with people withdementia. It is important for the performers to learn the essence of the work and act
authentically in order to get that essence across.” I believe that it is important tostrive to constantly touch the hearts of older people with dementia and to givethem pleasurable emotions.

Bibliography

52

Evaluation wisdom in practiceEvaluation wisdom in practice6

P. 70 ⑤⑥

53

Summary
These evaluation initiatives represent just someof the many types of evaluation approaches
in practice today. Great efforts were madet o set goals and benchmarks for evaluationin collaboration with a variety of stakeholders.
Evaluation methods and detailed measures werecreated, tailored to the goals of the given project.
In sum, evaluation experts help you in:࠮ Designing an evaluation method and indicatorsthat reflect the opinions of stakeholders
࠮ Designing sophisticated surveys and analysing
their results࠮ Offering advice and guidance from an outside
standpoint
Evaluations take time, but we hope that you
have found some ideas here to help you along
the way.

Communication throughevaluation
Evaluation can be a hub through which
participants can communicate from diverse
perspectives. We will examine key aspectsof effective communication through thefour stages of evaluation, which can thenenhance project operations and development.

54

55

Socially inclusive cultural projects can involve not only governments
and arts organisations, but intermediate support organisations,experts (in art management, social welfare, evaluation, and so on),participants, and residents. When people coming from different
perspectives work together to promote a project, evaluation canserve as a powerful tool by providing a common language. Thisfunction of evaluation is currently undervalued.
There is often confusion about evaluation because of its relationshipto policy. For example, arts organisations wish that evaluation could
demonstrate the social value of our projects to the government,
which would in turn lead governments to improve policy.

56

Policy:Medium-term plan of
4 to 5 years

Policy areas:

Focus areas of themedium-term plan

Administrative Programmes:
With a budget of about 3 years(cultural facility management
projects, subsidy projects, etc.)

Arts organisations

Although there are some differences between local governments,
administrative evaluations are generally conducted along a
trajectory as shown on the right. At the top, “Policy” refers to a
medium-term plan of four to five years. “Policy areas” refer to focus
areas, which are activated through “administrative programmes.”
Administrative programmes last for approximately three years, thelength of a typical administrative budget. A project, which usually
includes a series of events, is conducted in a single fiscal year.

Layers of evaluation in public projects

Government

From the government perspective, though, the main point ofproject evaluations is to check whether budgets have been
administered correctly and whether projects have been carriedout as planned. Within such a framework, there is little potential
for policy change.

Some point out that government evaluations do not involve theevaluation of policies as a whole. Even if arts organisations takeevaluation very seriously and submit reports on projects, programmes,and events, they do not know how the information they submit isgoing to be used. As such, they may feel rather distant from thegovernment. In these circumstances, what are the possibilities for
project evaluations?

Communication through evaluation

Communication through evaluation

Evaluation of cultural projects in relation to policy* Based on our interviews
Abstract

Policy Objectives and Direction

Items in the mid-term plan

Budgetary unit

Project/Programme:
One year

Events:Events within a projectSpecific57

If we see evaluations as communication hubs, we can framethem as opportunities for diverse stakeholders to interact witheach other in novel ways. This has many potential benefits.
Arts organisations have a forum to convey the significance of
their projects in terms with which government representatives are
more familiar (e.g. quantitative data). These representatives arethen better able to explain the value of cultural projects in terms
that superiors, financial departments, and citizens can understand.
From page 60 on, we will discuss evaluation in four stages: evaluation
design, indicator review, data collection, and results utilisation(see the figure on the right). At each stage, we will examinesome of the perceptions of evaluations from government andarts organisation perspectives and share key points on how to
use evaluations as opportunities for communication.

Evaluation stages that provide opportunities for communication

Government

We must conduct thoroughevaluations to set budgets,
but is it enough to just evaluate
cultural projects?

Four stages of evaluation→

Evaluation
designIndicator
review

Datacollection

Results
utilisation

Designing the evaluation
goal, methods, schedules,
and budgets in line withthe project goal

Considering and crafting
the indicators used toverify project activities

Gathering a variety ofdata on the content
and results of project
operations

Analysing the data
and utilising results
strategically

The remarks are based on 18 interviews with staff members ofgovernments, arts organisations, and intermediate organisations,as well as researchers and experts in evaluation and culture.
Although some cultural projects are hosted by the government, we will
focus on the projects subsidised or commissioned by government.

I want the government to knowwhat comes up on-site as well.
I wish I could conduct evaluations
that get this across.

Artsorganisations

58“Government” here includes officials from ministries,
prefectural and municipal administrations,and other cultural departments.

Communication through evaluation

Communication through evaluation

Communication in four stages of evaluation“Arts organisations” here refers to people in the field
carrying out socially inclusive arts activities.

59nice!

Designing the evaluation goal, methods, schedules, and budgets in linewith the project goal

How should cultural projects
be evaluated?

All I can come up with areparticipant numbers andsatisfaction surveys.

Evaluation design is an opportunity todevelop a circle of people with different
perspectives who will work toward the
project goals together.

I wish arts organisations
would put as much care
into their self-evaluations
as they do their projects.

0\ERVVDQGWKHƓQDQFH
department are asking me
to justify these projects withnumbers. I also have to conduct
proper evaluations so I can makebudget estimates for next year.
But how?

Government
Budget acquisition

Evaluation goals

Before selecting evaluation methods, consider
evaluation goals. The evaluation methods used
for project reports, project improvement, andadvocacy will differ depending on goals.

Evaluation design

Artsorganisations

When should evaluations be designed?

Think of evaluation as an integral part of the project as awhole. When planning a project, evaluations should bedesigned in accordance with overall project goals. Someintermediate support organisations provide effective ongoing“adjacent support”, giving advice to governments and arts
organisations from the project planning stage onwards.

We should conduct evaluation after
a project is complete, shouldn’t we?

Communication through evaluation

Communication through evaluation

Evaluation design

Setting the scene for
evaluation

Governments and arts organisations
must create opportunities for various
project stakeholders to gather andshare thoughts on both project andevaluation goals.
Although they may have disparate
interests and opinions, searching forconsensus through dialogue is key.

Once we set a clear-cut goalfor our project, we will be
evaluated only on whether or
not we achieve our goal.

Numerical data is often easier for financedepartments to comprehend. But when project
goals are clearly conveyed to funders, they are
more likely to understand the significance ofthe qualitative assessments beyond thenumbers.

Special qualities of culturalprojects and their evaluation
In the arts, situations often change after
initial project design. Rather than using
a particular set of measures from the
beginning, measures can be crafted tosuit current circumstances.

But there isn’t one “right” wayto do it in art, which is whatmakes it so valuable.

The participants are diverse and unique;does the government really believe that
we can evaluate the impact of a culturalproject on each person in the same way?

60

61

nice! significance and strengths of our projects

In the indicator review, putting theConsidering which indicators should be used to verify the results of
DSURMHFW·VSURFHVVHVDQGRXWFRPHV

I wish arts organisations
would use more quantitative
indicators in self-evaluation.

Government

into words can also mean being more
specific about project activities.

,WōVGLIƓFXOWWRVXPXSWKH
opinions when listening to
the diverse needs of people
involved in a project.

What indicators can be measured
numerically?

We tend to understand quantitative indicators
as outputs, e.g. number of participants, but it
is also possible to develop ways to quantify
outcomes.

Indicator review

Indicators can illuminate
project strengths

Articulating our project strengths in theform of indicators allows us to engage inthe process of naming the significance ofthe project to ourselves.

Artsorganisations

What on earth is an indicator?
I don’t normally have to think
about this to get things done.

The positive social impacts of socially inclusive arts
projects extend not only to the target populations,but to their family members, to management staff,
and often to the broader community. Shouldn’t wecall those changes project results, too?

62

Creating opportunities forindicator review

Participants, artists, and staff, all comingfrom diverse perspectives and fields ofknowledge, should be involved in thevital process of creating indicators to
discuss what impacts they would like theproject to have.

What aspects can beconceptualised asindicators for socially
inclusive cultural projects?

Communication through evaluation

Communication through evaluation

Indicator review

Approaches to creating
indicators
First, look around you. Whatcan be gathered through all of
the data, perhaps in the formof day-to-day documentation,that you already have?

Look at your target group

Take an approach to creating indicatorsthat has the potential to promote thestrengths of all involved.

In activities with older people, thefocus is often on what people cannot
do. I’d rather learn to create activities
from the perspective of discovering
what people can do and making themost of that.

63

of your project based on specific data,rather than your personal feelings.

Gathering data on the content and results of project operations in various
ways, such as through questionnaires, interviews, observations, andconversations with participants,WōVGLIƓFXOWWRVHHWKHUHVXOWVRI
a cultural project in just a year.

Government

Gather data through a diversity ofmethods, not just one.

Data collection

Artsorganisations

How do I explain the datathat you have gathered?

There are limits to what you canget from a questionnaire.

What data will you collect?

64

nice! This helps you to explain the significanceDocument the project on a daily basis(journaling, minutes, photos, etc.) to keep
track of the process, especially the unexpecteddevelopments, and archive these materials.
They may become useful later.

Span of data collection

Determining the meaning of
the data

Some results emerge over time, as
evaluations are conducted over thecourse of years. The time frame of publicprojects needs to be reconsidered fromthis point of view.

:KHWKHU LW·V TXDQWLWDWLYH RU TXDOLWDWLYH
you need to be able to explain the context
of all your data.

Partnerships with evaluation experts
Partnerships with experts and intermediate support
organisations may bring meaningful results inevaluation. In some cases, experts especially
support the process of creating indicators andcollecting data on outcomes that might seemdifficult to turn into data.

It’s better to survey
audience members at theevent anyway, isn’t it?
How are the evaluators deciding
what’s good and bad about it?

Communication through evaluation

Communication through evaluation

Data collection

Art data

Figuring out how to gather information
on arts activities that participants might
say cannot be put into words can in
itself be creative work. We get a senseof this when we examine not only thequality of the work, but the ways that art
transforms people and situations.

What data should
be used to evaluatesocially inclusive art?

65

nice! Reporting evaluation results in a way

Government,W·VJUHDWWRKDYHDZULWWHQ
report as the basis for
explanations to my bosses
and the finance department.
Reports have also been madeavailable to the public on thegovernment website.

Strategic use of results

Has evaluation and report writing becomeD JRDO LQ LWVHOI" ,W·V HVVHQWLDO WR FUHDWH D
strategy for how evaluation results anddata will be used.

Utilisation of results

Speaking to your audience
Who is it most important for you tocommunicate your results to?

The content you choose, the way you organise thecontent, and the use of illustrations should all work
to support the reader to understand with ease.

Artsorganisations

66

that can reach diverse audiences is apart of advocacy.

The stage of utilizing the results of evaluations in order to improve project
designs and managements, and to communicate the value of culturalprojects to citizens.

The benefits of ouractivities are hard to
express—especially to thegovernment.

,·PQRWUHDOO\VXUHLIWKH
content of this project isa good fit for current
social conditions.

Evaluating to quit!?

Deciding to stop doing something
as a result of evaluation can beseen as a way to foster project
renewal and new development.

Enabling project renewal
and development
Evaluation is a cycle. When aproject ends, the people involved
should reflect and think about
how to improve and develop.

Even when we report to thegovernment, their responses
come late or not at all. I wish
we could get their feedback
so that we could use it toimprove our projects for the
coming year.

I want to convey thevalue of cultural
SURMHFWVLQDZD\WKDW·V
easy for citizens tounderstand.

Communication through evaluation

Communication through evaluation

Results of utilisation

Communication hub

Evaluation is a tool that provides a common
language between various stakeholders
in a project, e.g., government and arts
organisations, government and citizens,one government department and another,and organisations and experts.
Training people to communicate the valueof cultural projects effectively can lay thegroundwork for promoting activities thatwork towards an inclusive society.

I want to get others tounderstand thesignificance of ouractivities, to foster acommunity oflike-minded people wecan collaborate with.

67

Summary
How does evaluation in socially inclusive culturalprojects differ from evaluation in other fields? Theremay not be major differences in the practical work of
the evaluation process itself. When envisioning
project outcomes, though, what often comes tomind are powerful observations of participants’behavioural shifts. The term “evaluation” can bedaunting, like it requires an “objective” interpretationof data gathered from a huge group of people. But
raising awareness of the positive changes on an
individual level may support staff, facilitators, andothers working on the project to avoid potential
pitfalls in subsequent projects.
In this chapter, we discussed how evaluation could
be better understood as a hub (or node) for communication between various project stakeholders. We
covered points that could be of help in project
management, improvement, and development,
focusing on the relationship between government
and arts organisations. Connecting and cooperatingwith people from different perspectives and fields
could be a great boost to socially inclusive arts
activities as a whole. We hope that these ideas
help you to envision evaluation as a powerful act
between the diversity of people involved in a project.
Evaluation through dialogue provides an opportunity
for people to develop a common language, supporting them to solve problems and to work collectively
toward a truly inclusive society.

68

Interviewees
We would like to acknowledge the individuals who responded through interviews
and study sessions conducted between May 2019 and February 2020. Affiliations
and positions were accurate at the time of interview.

OKABE TaroManaging Director, Tanpopo-No-Ye Foundation

OCHIAI Chika

Chief Operating Officer, K-three Inc.

ONIKI Kazuhiro

Director for Facilities, Culture Promotion Division (chief researcher),Culture and Tourism Bureau, City of Yokohama

KAKITSUKA Takuma

Toyonaka Performing Arts Centre, Japan Century Symphony Orchestra

KATAYAMA Masao

President, The Saison Foundation

KUSAKA Nahoko

7YVMLZZVY-HJ\S[`VM°*VU[LTWVYHY`:VJPHS:[\KPLZ+VZOPZOH>VTLUZ°*VSSLNLVM3PILYHS(Y[ZSUGIURA Mikio

Programme Director, Arts Council Niigata, Niigata City Art & Culture Promotion Foundation

SUGIZAKI Eisuke

Yokohama Arts Foundation (a public foundation) PR & ACY GroupNAKANISHI Miho

Chief Executive Officer, Osaka Arts Council

FUJII Masahiko

Clinical Professor, School of Medicine, Tohoku University

FUJIOKA Yasuhiro

Associate Professor (Architecture Seminar), Institute of Urban Innovation,
Yokohama National University

MIURA Hiroki

Advisor, Oita Prefecture Arts, Culture and Sports Promotion Foundation;
Director of Research Department, Oita Association of Corporate Executives

MINAMOTO Yuriko

Professor, Graduate School of Governance Studies,
Meiji University; Director, Institute of Program Evaluation

YUASA Manami

Head of Arts Japan, British Council

YOSHIDA Takayuki

Associate Professor of Urban Policy, Graduate School of Urban Management,
Osaka City University

YOSHIMOTO Mitsuhiro+PYLJ[VY*LU[LYMVY(Y[ZHUK*\S[\YL5309LZLHYJO0UZ[P[\[L

Two members of the Agency for Cultural Affairs of Japan also participated kindly.

69

Bibliography
7SLHZLUV[L[OH[<93ZHYLHJJ\YH[LHZVM-LIY\HY`

■J a p a n e s e L i t e r a t u re①20:/04HZHOPRV0:/062(;VTVUVYP4(9<@(4(:H[VTP:OP[Z\[LRPZOHRHPJOȬZHUVOȬOȬ!;HZOHUVNȬYPZLPUVYPRHPZOHRHPNHR\8\HSP[H[P]LYLZLHYJOTL[OVKVSVN`!0U[LYWYL[P]LZVJPVSVN`VM[OLYH[PVUHSP[`VM[OLother). Yuhikaku.
②:(;ǰ0R\`H-ȟY\KV^ȊR\UVNPOȬɬ[VPVZVKH[LY\RHZL[Z\VRP[HLY\-PLSK^VYR[LJOUPX\LZ!-VZ[LYPUN
questions and raising hypotheses). Shinyosha.
③5(2(4<9(4PH .LPQ\[Z\RH[Z\KȬUPVRLY\R`ȬZȬUVZHPRȬɬZȬaȬ[VLUWH^HTLU[VUV[Z\UHNHYPVZHN\Y\9LJVUZPKLYH[PVUVM*VJYLH[PVUPU7HY[PJPWH[VY`(Y[Z!,_WSVYPUN[OL3PURIL[^LLU*YLH[P]P[`HUK
Empowerment). Cocreationology 1, 31-38.
(https://nihon-kyousou.jp/cocreationology/vol1_no1/Cocreationology_1-1-6.pdf)④ Fetterman, Wandersman, et al., (2014) Empowerment evaluation principles in practice. Japanese edition,,UWH^ȊTLU[VO`ȬRHUVNLUZVR\[VQPZZLU!2`ȬPR\M\R\ZOPPY`ȬRPN`ȬRVT`\UP[PRHPU`ȻW\YVN\YHT\UVkaizen to kassei-ka ni mukete. Translation supervised by SASAO Toshiaki, translated by TAMAI Kouta andOOUCHI Junko. Kazama Shobo.
⑤-<1004HZHOPRV4(,+(@\\ZHR\2(5,+(,YPRV:(:(20/PKL[HKH5PUZOPZOȬQȬKȬY`ȬOȬ+LTLU[PH
emotional therapy). Horinsha.
⑥-<1004HZHOPRV:(:20/PKL[HKH¸5PUZOPZOȬ^HJOPY`ȬRHUȬUHZOPRRHURH&ɬ)7:+UVQȬKȬY`ȬOȬRHYH
TP[HRȬZH[Z\¹0ZKLTLU[PHH[YLH[HISLKPZLHZL&5PWWVU9VULU0NHRRHPAHZZOP1HWHULZL1V\YUHSVM
Geriatrics) 54(2), 114-118.
(https://www.jstage.jst.go.jp/article/geriatrics/54/2/54_114/_article/-char/ja/)⑦ HORI Kimitoshi. (2004) Fashiriteshon Nyumon (Introduction to Facilitation). Nikkei Business Publications.
⑧405(46;6@\YPRV:HURHNH[HO`ȬRH!2HPaLU[VOLURHR\UV[HTLUVO`ȬRHUVQPZZLU7HY[PJPWH[VY`
evaluation: Practice of evaluation for improvement and change). Koyo Shobo.
⑨@(:<+(;VTV`\RP7\YVN\YHT\O`ȬRHɬ[HPQPURVT`\UP[PLUQVUVZOP[Z\V[HRHTLY\[HTLUP7YVNYHT
evaluation: Improving the quality of interpersonal and community support). Shinyosha.
⑩@(4(<*/0@\\OLP46909LPUH(5A(0@\\RP>ȊR\ZOVWW\KLaHPUYVU>VYRZOVW+LZPNU2LPVUniversity Press.
⑪96::07L[LY/LUY`307:,@4HYR>-9,,4(5/V^HYK,,]HS\H[PVU!(Z`Z[LTH[PJHWWYVHJO
1HWHULZLLKP[PVU7\YVN\YHT\O`ȬRHUVYPYVU[VOȬOȬ!:OPZ\[LTH[PRR\UH[HPQPUZȊIPZ\ZLPZHR\O`ȬRHUVQPZZLUNHPKV[YHUZSH[LKI`66:/04(0^HV/09(62(2ȬJOP4690;VZOPV46;65(.(;HR\YȬ5PWWVU/`VYVUZOH
⑫>,0::*HYVS/,]HS\H[PVU!4L[OVKZMVYZ[\K`PUNWYVNYHTZHUKWVSPJPLZ1HWHULZLLKP[PVU5`ȻTVU
O`ȬRHNHR\!:LPZHR\W\YVN\YHT\RLUR`ȻUVOȬOȬ[YHUZSH[LKI`4(,2(>(4PRV02,+(4P[Z\Y\:(:(20
9`Ȭ5PWWVU/`VYVUZOH

■E n g l i s h L i t e r a t ure⑬ Belfiore, Eleonora. (2002) Art as a means of alleviating social exclusion: Does it really work? A critique of instrumental
cultural policies and social impact studies in the UK. International Journal of Cultural Policy, 8 (1), 91-106.
⑭ Belfiore, Eleonora. (2010) Beyond the “Toolkit Approach”: arts impact evaluation research and the realities of
J\S[\YHSWVSPJ`ɫTHRPUN1V\YUHSMVY*\S[\YHS9LZLHYJO

70⑮ Belfiore, Eleonora and Oliver Bennett. (2008) The Social Impact of the Arts: An Intellectual History.
Basingstoke: Palgrave Macmillan.
⑯ Blanche, Rachel. (2018) Redefining notions of quality in participatory arts. Arts Management Quarterly, 128, 12-19.
⑰ Contandriopoulos, D., & Brousselle, A. (2012) Evaluation models and evaluation use. Evaluation, 18(1), 61–77.
⑱ Matarasso, Francois. (2013) Creative progression: Reflecting on quality in participatory arts. UNESCO
Observatory Multi-Disciplinary Journal in the Arts, 3 (3), 1-15.
⑲ Meyrick, Julian, Robert Phiddian and Tully Barnett. (2018) What Matters? Talking Value in Australian Culture.
Melbourne: Monash University Publishing.
⑳ Meyrick, Julian, Tully Barnett and Robert Phiddian. (2019) The conferral of value: The role of reporting
processes in the assessment of culture. Media International Australia, Incorporating Culture & Policy, 171 (1),80-94.

■P roject r epor ts㉑(Y[Z*VUZVY[P\T6P[H¸/LPZLPULUKVȊ[Z\RVUZȬZOPHT\ǰP[HRȬJOPR\RLPRHR\QPZZLRPOȬRVR\ZOVGI\URH[VO`ȬRHOHUKVI\RR\¹(Y[Z*VUZVY[P\T6P[H*VUZ[Y\J[PVU7SHU7LYMVYTHUJL9LWVY[¸*\S[\YLHUK
Evaluation Handbook”)(https://www.pref.oita.jp/uploaded/life/1056591_2471134_misc.pdf)㉒2P[HR`\ZO\(Y[Z;OLH[LY¸1PN`ȬO`ȬRHJOȬZHOȬRVR\ZOV¹)\ZPULZZ,]HS\H[PVU:\Y]L`9LWVY[(http://q-geki.jp/aboutus/history.html#report)㉓2[OYLL0UJJVS\TU¸:OȬNHPZOHUP`VY\NLPQ\[Z\I\URHRH[Z\KȬRHYHºIHYPHM\Yȟ»VRHUNHLY\¹*VUZPKLYPUN[OL
meaning of “Barrier Free” in arts and cultural activities by people with disabilities)(https://note.com/k_three/n/n86f3bd5bec31）㉔;HUWVWVUV`L-V\UKH[PVU+PZHIPSP[`HUK(Y[Z*VUZ\S[H[PVU¸:OȬNHPUVHY\OP[VUVȊ[V[VO`ȬRHHUH[HUVºTVUVZHZOP»RPRHZL[LR\KHZHP¹(Y[HUKL]HS\H[PVUVMWLVWSL^P[OKPZHIPSP[PLZ!;LSS\ZHIV\[`V\Y¸TLHZ\YLZ¹
(https://artsoudan.tanpoponoye.org/wp/wp-content/uploads/2017/05/monosashi_web.pdf)㉕1HWHU-V\UKH[PVUMVY9LNPVUHS(Y[(J[P]P[PLZ¸2ȬYP[Z\I\URHZOPZL[Z\UPVRLY\ZLPZHR\O`ȬRH[ȬUVHYPRH[HUP
RHUZ\Y\JOȬZHRLUR`Ȼ·RȬR`ȬOȬY\RȬYP[Z\NLRPQȬUVO`ȬRHZOPZOPU¹9LZLHYJOVUWVSPJ`L]HS\H[PVUILZ[practices in public cultural facilities: Evaluation guidelines for public halls and public theaters)(https://www.jafra.or.jp/library/report/18/）㉖;VR`V)\URH2HPRHU¸2ȬYLPZOHT\RLVUNHR\^ȊR\ZOVWW\UVRLUZOȬB/LPZLPULUKVD¹,]HS\H[PVUVMT\ZPJ
workshops for elderly people, 2018)(https://www.t-bunka.jp/cms/wp-content/uploads/2019/06/180314_Senior-ws_japanese.pdf)㉗(NLUJ`MVY*\S[\YHS(MMHPYZVM1HWHU¸)\URHZLPZHR\UVO`ȬRHZO\OȬUPRHUZ\Y\JOȬZHRLUR`Ȼ¹9LZLHYJOVU
cultural policy evaluation methods)(https://www.bunka.go.jp/tokei_hakusho_shuppan/tokeichosa/bunka_gyosei/pdf/bunka_houkoku.pdf)㉘9`\R`\7OPSOHYTVUPJ6YJOLZ[YH¸)\URHJOȬZO\ZHPIHYPHM\YȟRȬLUºJO\YHZH\Ua\RVUZȊ[V»RHPZHP¹)HYYPLYMYLL
performance “Chura Sounds Concert” sponsored by the Agency for Cultural Affairs of Japan)(http://bit.ly/2Fnz1cX)

71

Closing Remarks

The Agency for Cultural Affairs of Japan & Kyushu University

American historian Jerry Z. Muller writes:
There are things that can be measured. There are things that are worth measuring.
But what can be measured is not always worth measuring; what gets measured may
have no relationship to what we actually want to know. The costs of measuring may
be greater than the benefits. The things that get measured may draw effort awayfrom the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive. (The Tyranny of

Joint Research Team Members
The Agency for Cultural Affairs of Japan
Yuki Asakura

Research Officer, Policy Coordination and Policy Research Group,
Headquarters for Vitalizing Regional Cultures, Agency for Cultural Affairs of Japan

Masaru Aoshiba

Group Sub-leader, Policy Coordination and Policy Research Group,
Headquarters for Vitalizing Regional Cultures, Agency for Cultural Affairs of Japan

Terumi Okuda

Chief, Policy Coordination and Policy Research Group,
Headquarters for Vitalizing Regional Cultures, Agency for Cultural Affairs of Japan

Muller does not reject evaluation in general. “The problem is not measurement,
I\[ L_JLZZP]L TLHZ\YLTLU[ HUK PUHWWYVWYPH[L TLHZ\YLTLU[·UV[ TL[YPJZ
but metric fixation.” (p. 4) So, is it possible to carry out evaluation that, rather
than shrinking cultural activities, leads to increased creative activity?

Reiko Nagai

Staff, Policy Coordination and Policy Research Group,
Headquarters for Vitalizing Regional Cultures, Agency for Cultural Affairs of Japan

How can evaluation help us make the social significance of cultural projects
more tangible? To find answers to these questions, we have conductedsurveys on overseas trends, interviews, open research meetings, and otheractivities, and have held a number of discussions. Guides on evaluationTL[OVKZOH]LHSYLHK`ILLUW\ISPZOLKI`;VR`V(Y[9LZLHYJO3HI[OL:VJPHS
0TWHJ[ 4HUHNLTLU[ 0UP[PH[P]L (Y[Z 576 3PUR HUK ZV VU ;OPZ OHUKIVVR
therefore focuses on methodologies, such as approaches to evaluation andevaluation design, rather than methods of evaluation. We hope that you will
use this handbook along with the previous volume, Social Inclusion ThroughCulture and Art: A Handbook for Beginners.

Mia Nakamura

Associate Professor, Kyushu University Faculty of Design
3LHKYLZLHYJOLYH\[OVYVM*OHW[LY

Yuichiro Nagatsu

Assistant Professor, Kyushu University Faculty of Design(Author of Chapter 2)

Tsukasa Muraya

Research Fellow, Kyushu University Faculty of Design(Author of Chapter 4)

Tabihito Fujihara

Technical staff, Kyushu University Faculty of Design

As with the first handbook, in preparing this one, we received much useful
information and many valuable suggestions from countless individuals in ourinterviews and open workshops. I would like to take this opportunity to
express my sincere gratitude.

Torao Osawa

3LHKYLZLHYJOLY(Y[ZHUK*\S[\YL7YVQLJ[Z6MMPJL5309LZLHYJO0UZ[P[\[L

Satoshi Miyata

Office Manager, donner le mot (NPO) (Author and editor of Chapter 3)

Metrics, Princeton University Press, 2019, p. 3.)

Kyushu University Social Art Lab

Advisers

Secretariat

Mia Nakamura

3LHK9LZLHYJOLY[OL(NLUJ`MVY*\S[\YHS(MMHPYZVM1HWHU 2`\ZO\<UP]LYZP[`1VPU[9LZLHYJO;LHT

Kana Sakurai

Staff, donner le mot (NPO) (Author and editor of Chapter 3)

Associate Professor, Faculty of Design at Kyushu University

72

73

Social Inclusion throughCulture and the Arts:

A Handbook for Evaluation
Creating new values for diverse individuals in our society

Date of Issue:

23 March 2021 (Original Japanese edition: 25 March 2020)

Compiled by:

The Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team

3LHKYLZLHYJOLY!4PH5HRHT\YH

Authors:

Mia Nakamura, Yuichiro Nagatsu, Tsukasa Muraya, donner le mot (NPO)English translation:

Emma Parker Supervising

English editing:

Yaya Yao, Mia Nakamura

Editing:

donner le mot (NPO)

Design:Kaori Nagasue

Co-sponsors:

Fukuoka City Foundation for Arts and Cultural Promotion

Supported by:

Japan Association for Arts Management, Kyushu BranchWith the assistance of:

FY2019 Project to Promote the Arts and Culture at Universities,
the Agency for Cultural Affairs of Japan

0ZZ\LKI`!:VJPHS(Y[3HIHMMPSPH[LK^P[O[OL-HJ\S[`VM+LZPNU2`\ZO\<UP]LYZP[`
4-9-1 Shiobaru, Minami Ward, Fukuoka, 815-8540, Japan
http://www.sal.design.kyushu-u.ac.jp/

࠮;OLJVU[LU[ZVM[OPZOHUKIVVRHYLIHZLKVU[OLYLZ\S[ZVM[OL 7YVQLJ[MVY1VPU[9LZLHYJO7YVQLJ[IL[^LLU[OL(NLUJ`MVY*\S[\YHS(MMHPYZHUK<UP]LYZP[PLZ9LZLHYJO0UZ[P[\[LZL[J ¸:VJPHS0UJS\ZPVU;OYV\NO[OL(Y[ZHUK*\S[\YL¹
࠮;OPZOHUKIVVRPZH]HPSHISLMVYKV^USVHKH[O[[W!^^^ZHSKLZPNUR`\ZO\\HJQWLUNSPZOW\ISPJH[PVUZO[TS࠮;OPZOHUKIVVRTH`IL\ZLKYLWYPU[LKJVWPLKZOHYLKL[JMVYUVUJVTTLYJPHSW\YWVZLZVUS`HZSVUNHZ[OL
source is cited.
© 2021 The Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team