Social Inclusion Through Culture and the Arts: A Practical Evaluation Handbook The Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team

Let’s do it!

Social Inclusion ThroughCulture and the Arts:
A Practical Evaluation
Handbook
By The Agency for Cultural Affairs of Japan& Kyushu University Joint Research TeamThe voices ofmany come together
to create new values
in our society

S O C I A L I N C L U S I O N × C U LT U R E A N D T H E A R T S

Let’s do it!

Social Inclusion ThroughCulture and the Arts:
A Practical Evaluation
Handbook
By The Agency for Cultural Affairs of Japan& Kyushu University Joint Research TeamThe voices ofmany come together
to create new values
in our societyContents

Introduction

The extent to which we are able to take a flexible approach to theevaluation of cultural projects determines whether or not such
evaluation will be meaningful. Standard evaluation practices follow a
process of determining project goals and targets at the outset andthen checking whether these have been achieved. They give theimpression that if we simply follow the steps which have been set out,
just like an easy recipe or a manual for assembling a piece offurniture, we will be able to do it well.
With cultural projects, however, even though long-term goals do notc h a n g e , s h o r t - t e r m t a rg e t s m a y a l t e r i n t h e c o u r s e o f t h e p ro j e c t .
Moreover, the initial setting of targets is often difficult for socially
i n c l u s i v e a r t s a c t i v i t i e s . S u c h c a s e s re q u i re a p ro c e s s o f f l e x i b l e
adjustment of project targets and of the subjects of evaluation as we
carry out the project and its evaluation.
This handbook aims to shed some light on this flexible process. The
opening worksheets may seem a little complicated compared withthose in standard evaluation textbooks. However, they are based on
the experiences of stakeholders who are making meaningful use ofe v a l u a t i o n a s t h e y c a r r y o u t c u l t u r a l p r o j e c t s . O rg a n i z i n g t h e s e
p r a c t i c e s i n t o t h e p ro c e s s o u t l i n e d i n t h i s h a n d b o o k t o o k a g re a tamount of time and effort. We hope that you will put it to good use,
and if you have any suggestions for improvement, we would be
grateful for your input.

Introduction

2

Worksheets

4

Evaluating the unique nature of cultural projects

11

Taking a peek at the evaluation process

23

Four case studies

35

Dialogue: The first step on the evaluation journey

65

List of members of the expert committee / interviewees

79

Reference materials

80

Conclusion

82

Research team members

83

This handbook is a sequel to “Social Inclusion Through Culture andthe Arts: A Handbook for Beginners” and “Social Inclusion ThroughCulture and the Arts: A Handbook for Evaluation.” We hope you willrefer to these as appropriate.

Mia Nakamura

Lead Researcher, the Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team
Associate Professor, Faculty of Design at Kyushu University

2

3

WORKSHEET

Download and print out the worksheets here:
http://www.sal.design.kyushu-u.ac.jp/publications.html

These worksheets are intended for you to use and adapt to your own evaluation process.

Use them to discuss the project with the various people involved. Even if your project has
already begun, you can make use of these worksheets to reflect on the process to date.

01 Worksheet

02 Worksheet

Evaluation design

Evaluation design

Title of the cultural project:

See Chapter 1,
p. 14 – 16

See Chapters
2&4

Identify all of the stakeholders involved

Clarify the goals and targets

Write down everyone involved with carrying out this project. Making a list will help
you understand the current communication gaps, and what changes are needed.

Put the goals and targets of the cultural project into words.
This may be difficult to do off the top of your head, but it is important to write
them down at least once.

1. Write the names of all the people you can think of on the “Map of people involved,” as wellas the organizations or groups to which they belong.
2. Draw lines to connect the people you are already communicating with.

Goals

Goals

What changes do you hope to achieve in society throughthis cultural project?

E.g. Society respects the unique individuality of people with dementia

3. Mark with a ★ the people whose opinions you want to learn more about.
4. Mark with a ♥ the people you want to talk to as the project is underway or during reviews,
to make improvements.

Right

Left

［Stakeholder Map］

yRopewad
Sea RorTargets

Government

What do you hope to achieve through the cultural project
this year, in concrete terms?

the sections in ministries or
local governments involved
with cultural, welfare, or
education projects, etc.

E.g. The creation of situations which draw out par ticipants' unique

Specialists

fields related to the project, e.g.,
evaluation, clinical psychology,
welfare, art, education

Intermediary organisations

arts councils, foundations promoting thearts and culture, etc.

individualit y through ar ts activities

Which objectives willlead us to our goal?

ice
adv s!
ple
Sim ginnerbeforTargets

BEFORECHECK!

AFTER

Are the targets focused on the changes coming directly from the artistic activity?
Do you have a concrete picture of how to achieve the targets, or of what achieving
them would look like?

4

Supporting organisations

Participants understand the fun nature of art

Through art, participants develop theirunderstandings of the differences between
what each of them experience as interesting and fun.

Artists

project organisers – individuals or groups

Key target population

participants

Other supporters

I don’t have a clear idea of evaluation...

Socially inclusive value

Socially inclusive artistic value

People / organisations implementing the project

Families of project
participants

It is important to set targets in the zone where “artistic value” and “socially inclusive value”
overlap (see diagram below). To set appropriate targets for evaluation, note the following points.

Artistic value

Coordinators / facilitators

Funders

Each route has its positives.

Partner organisations
Event participants

sign language groups, information
accessibility apps, venue owners, etc.

Operational staff
Support staff

Feel free to add people
who do not belong to anyof these categories.
It is also fine to leavesome categories blank.

NPO staff in direct contact with participants,
professional carers, school staff, etc.

Tick the following boxes for the targets you wrote on the left, as well as the mapof people involved.
CHECK!

Are all the people needed to achieve the targets set?
→ Talk this over with the people involved.

Are the key needs of the people listed reflected in the targets?

→ If they aren’t, look over the targets once more with the people involved.

5

Download and print out the worksheets here:
http://www.sal.design.kyushu-u.ac.jp/publications.html

03 Worksheet

04 Worksheet

Data collection

Data collection

What types of records do we need?

See Chapters
3&4

What materials might need to be kept as a record of project achievements?
●Circle the types of records which you normally keep when carrying out a project,●or which you think you could keep.

Minutes

Event record

Interviews

Publicity materials

Photos/Videos

Other (

Make a plan for your data collection: Who will you collect it from, when, and how?
Think about the project flow in order to make a concrete plan and timeline.

staff

Questionnaires

Media coverage

Feedback from participants or local residents

Archive of works）●Think about what kinds of data in your records have been useful in tracking●
changes and impact, and make a note of these.

E.g. ‒ Not only photographs, but also videos, including videos of participants before andafter taking part in the event‒Pay particular attention to changes in peopleʼs expressions when taking photos‒Keep an archive of the works made by participants

participants

After implementation●What would you like to start recording?
●other stakeholders

During implementation

E.g. ‒ Frequency with which key words appear in activity logs or interviews‒ Unique episodes demonstrating change as reported by participants, families,
local residents, etc.

See Chapter
3

Before project implementation

Activity log

Plan your records: Who, when, and how?

Other

We already have quite
a lot of records that
we can use. Phew...

Ask specialists to help in recording
things that the group carrying out
the activity can’t manage.

Refer to Chapter 3 as you fill this in.

The same goes for securing a budget!

6

7

Download and print out the worksheets here:
http://www.sal.design.kyushu-u.ac.jp/publications.html

05 Worksheet

06 Worksheet

Disseminating results

Disseminating results

Review and reflect

See Chapters3 & 4Look back at the map of people involved to see who (outside your organization) you
will review and reflect on the project with.
Review and reflect on the project, referring to your records.
●What changes have you observed since the project was implemented?
●
E.g. The workshops brought out the unique individuality of people with dementia,
giving their professional carers and families a fresh perspective.

See the section on
evaluation, p. 26

Prepare to report on your projectDiscuss the target audience for your report, its focus, and how best to lay out your story.
●What media will you use for your reporting? Circle any applicable.
●
Report to funders

Hard copy report (booklet)

WebsiteOther ()

Copy or print this worksheet and answer the questions separately for eachof the media you will use.

●Write down anything that struck you in the feedback from participants or●others involved.

●What would you like to emphasise in your messaging to society or funders?
●●Who do you want to tell?
●
E.g., People who wanted to participate but could not, people who are thinking ofcarrying out similar activities in the future

Prioritise what will speak toyour primary target
audience.

●What will be the focus of your message?
●
E.g., Changes in the participants, increase in the number of partner organisations,
unexpected results, etc.

●What are the areas for future improvement?
●●Write down ideas for how you could convey your message.
●
E.g., Include many photos of participantsʼ faces in the report booklet

Make use of participants’ words
or points that emerged from
evaluation in your advocacy,
or in setting new indicators.

8

If you can report effectively online,e.g., on your website, it will help
your publicity and future activities!

9NotesThis handbook contains links to the two previously published handbooks, as well as to other reference materials. Please refer to these as
appropriate.
Social Inclusion Through Culture and Arts Handbook Series“Social Inclusion Through Culture and the Arts
– A Handbook for Beginners”＞ a primer, p. 00

Evaluating the unique natureof cultural projects
Evaluation encourages communication between thevarious parties involved with cultural projects andis an important tool for drawing out their unique value.

“Social Inclusion Through Culture and the Arts
– A Handbook for Evaluation”＞ information on evaluation , p. 00

However, if generic forms of project evaluation are
applied to cultural projects as they stand, the process
frequently does not work well. In Chapter 1, we willgo over these “evaluation pitfalls” and explore how
to avoid them.

The handbooks can be downloaded here:
http://www.sal.design.kyushu-u.ac.jp/english/publications.html
Referencematerials

10Pages 80 - 81

11

Evaluation BeginnerBut where should
I start...?

ally tricky,
This is re
isn’t it?

K?
he O
Is s

Second Year Colleague

Project Reports

You’re experienced at this!!
There are someconsiderations which are
unique to arts activities.
Let’s look at them before
we start.

Project Reports

What do you mean?

Well, fundamentally, when
we evaluate arts activities,
we need to focus on thecreation of new value!

Evaluate newly-created value
Cultural projects are different from the profit-oriented projects
carried out by businesses, or NPO projects which aim to solve
problems. They are value-creation projects, which emphasise
the creation of new value.
Let’s imagine you are climbing a mountain. Team X encounteredan unexpected storm but continued by the shortest route andwas able to reach the summit according to plan. However, theless skilled or less athletic team members had to give up
partway and return to the base. On the other hand, all themembers of Team Y retreated when the storm descended. Whenthey set out a second time, they took a detour to avoid fallentrees, but thanks to this they came across previously unknown
scenic spots and had a very rewarding experience.
How would you evaluate the two teams? If this were a profit-oriented or problem-solving project, Team X would probably beconsidered the most successful. However, if it were a value-creation project, Team Y, which generated unforeseen value andenabled its members to experience greater joy, would be seen
as more effective.

Evaluation suited to the unique nature of cultural projects

Critique

Policy Evaluation

Operational Improvements

Advocacy Validation of Results

Operational Improvements

Let’s tackle it according to our goals!
Our goal is a project report, so...

Project Reports

Evaluation suited to the unique nature of cultural projects

Key points

Cultural projects are value-creation projects. It is important to
run a project efficiently, with care regarding budget, but it is also
very important to recognize the new value which emerges. Whenevaluating socially inclusive cultural projects, in particular, it isnecessary to pay attention to the question of whether or not thecreation of value leads to problem-solving.
＞ information on evaluation, p. 20 - 21

12

13

Point 1

Ascertaining the public value of arts activities

Evaluation pitfall ❶Ms A wanted to emphasise the socially
inclusive results of her cultural project.
She learned evaluation methods fromNPOs supporting disabled people andpeople living in poverty, made a logic
model from the project planning stage,
and carried out an evaluation of thesocial impact. However, at the project
reporting meeting, the response was:“This isn’t a cultural project–it’s a social
welfare project, isn’t it?”

CultureSocial
WelfareIn evaluating socially inclusive cultural projects, how do thechanges which the cultural project has the potential to directly
bring about and the things which are necessary to make social
inclusion a reality overlap? This area of overlap is what is known
in economics and cultural policy as a project’s “public value.”
Cultural projects which lead to social inclusion have two kinds ofvalue: artistic value, such as stimulating participants in creative
new ways or nurturing their expressive skills, and socially
inclusive value, such as deepening their understanding of
marginalised groups, fostering a society which respects diversity.
But just because participants have built greater expressive skills,
for example, it does not mean that the project has improvedsocial inclusion: artistic value does not necessarily contribute toinclusion. Nor does simply implementing a cultural projectsuddenly bring about a society which respects diversity.

14

So it is necessary to identify which aspects of a cultural project
can directly foster social inclusion. For example, changes inperception, such as “I feel that I can do this, too,” or “I relate towhat others are feeling” are elements of a cultural project whichhave the potential to contribute directly to social inclusion. In
artistic activities, where there isn’t such a rigid boundary
between what is correct and what is not, participants can breakaway from their usual subject positions. Through freer forms ofexpression, they are able notice new aspects of themselves anddiscover new facets of other people. ＞ information on evaluation, p. 20
In this handbook, we call the parts of a cultural project whichhave the potential to contribute directly to social inclusion “thevalue of socially inclusive arts activities.”

Artistic value

Socially inclusive value

Evaluation suited to the unique nature of cultural projects

Evaluation suited to the unique nature of cultural projects

Let’s check the points about evaluation suited to theunique nature of arts activities and frequent pitfalls.

Value of socially inclusive arts activities

In general, arts activities are particularly effective at changing
people’s perspectives. ＞ a primer, p. 24
Such changes in individual perspectives can lead to changes in relationships and insociety. For example, a new understanding of one’s own abilities
can stimulate a zest for living, and increased empathy—thecapacity to relate to what others are feeling—can bring aboutchanges in interpersonal relationships. Changes like these canthen promote the social participation of marginalized groups anddiverse, inclusive communities.

15

Socially inclusive value
Vi t a l i t y

Re-evaluation of
one’s own abilities
Redefinition ofchallenging experiences
Ability to relate to thosedifferent from oneself
Understanding the connections
between personal experiences
and social issues
・・・・・

ved wellbeing
Impro

Changes in society
Promotion of the social
participation of minorities
Reduction in the numberof people struggling to cope
Realisation of a moredemocratic society
Betterrelationinterpersonal
ships

Participation insocial activities
・・・・・

Emergence of more diverse,
inclusive communities

Creating evaluation criteria which make thevalue of the activities visible

Evaluation pitfall ❷Kenta was told by his manager to demonstrate
the project results using numbers.
He included plenty of such data in the project
report, such as participant numbers, audience
numbers, and so on. But this information did notmeaningfully convey the project results. He wasextremely anxious that the project might be cut
based on this report, even though theparticipants were very happy with it.

Will the project be cut?

・・・・・

Value of socially inclusive arts activities

So, when evaluating socially inclusive cultural projects, it isimportant to stay aware of which elements of artistic activities
can lead to social inclusion.
This is important not only at evaluation time but when designinga project and setting project targets. If you make socially
inclusive value which is not directly created by the culturalproject into a project target, it will become more of a social
welfare project than a cultural project.

Discuss with your colleagues
what the value of a socially inclusivearts activity is for your team!

16

Point 2For cultural projects, high participant numbers and full venues
are not necessarily indicators of success. The idea of socially
inclusive cultural projects is to hold them with people with lessaccess to “mainstream” projects. It is important to implement
projects which satisfy the participants so much that they would
participate again, even if such projects do not reach a massaudience. The key here is to make the new values which werecreated on the ground (the unique changes) visible.

Evaluation suited to the unique nature of cultural projects

Evaluation suited to the unique nature of cultural projects

Artistic value
Changes in perspectives or relationships

There are various methods of doing this. One is to combine
quantitative data, expressed in numbers, with qualitative datasuch as feedback and anecdotes from participants and others
involved. While quantitative data is good at showing overall
trends, it is not good at conveying the unique changes which
took place on the ground—such changes are better illustrated
through qualitative methods. Combining the two makes it possible to convey the unique changes which took place on the
ground while also illustrating overall trends.

17

At music workshops for the elderly held at Tokyo Bunka Kaikan,
on the other hand, when the workshops were going well, participants beat out rhythms independently and became caught up in
the music without the artist facilitators taking the lead.
With this in mind, it might be＞ information on evaluation, p. 50 – 51
possible to judge the success of such workshops by observing
whether participants beat out rhythms independently or becamecaught up in the music.
In both cases, these indicators were observed when attention
was paid to the overlap between artistic value and socially
inclusive value, or “the value of socially inclusive arts activities.”

A more creative example of converting something into an indicator is the evaluation of YokohamArtLife’s project in the Sakonyama Housing Estate. The estate’s workshop staff recorded things
they “Saw,” “Heard,” and “Talked about” in a diary. By reviewing
and analysing the stories in the diary, they tried to measure theeffects in relation to their targets. (See Chapter 3)
Today’s widely recognized social and economic measures, suchas the GDP, IQ, wellbeing index, and so on were all created bypeople. These people creatively devised sets of indicators
based on their beliefs. Of course, it is impossible to replaceartistic value with words or figures, but it is possible, with a little
ingenuity, to turn some of the value of socially inclusive arts
activities into indicators.
As a note, Tokyo Bunka Kaikan and YokohamArtLife make use ofphotographs or illustrations in their evaluation reports for
aspects which are difficult to express in text or metrics, thus
making visible the unexpected impacts brought about by art.
Evaluation reports that effectively communicate a project’s
unique value serve advocacy efforts that support the project andthe broader goal of social inclusion throughout society.

Evaluation suited to the unique nature of cultural projects

Evaluation suited to the unique nature of cultural projects

Another method is to convert qualitative data into numericalvalues in your own original way. When we hear “numerical indicators,” we tend to think of existing indicators, but you can create
your own. For example, researchers at Sendai Tomizawa Hospital conducted observations on the changes brought about inelderly dementia patients through theatrical readings. They
noticed an increase in behaviours such as returning greetings,
expressing gratitude, and paying attention to others. They thenattempted to convert these into numerical indicators.
＞ information on evaluation, p. 52 – 53
At the same time as beingbrought about by the activation of emotional function throughthe arts activity, these behaviours also improve their relationships with caregivers and the care environment as a whole.

◀ Tokyo Bunka Kaikan,“Guidebook to the Art Activities leading to Social Inclusion”
https://www.t-bunka.jp/en/about/on_stage.html

T h e n um b er of gre et i n g s i n c rea s e d

Oh, I see!
Increases or decreases
in certain behaviours
can be an indicator.

◀ Yokohama Arts Foundation,“Report on ‘Leading Program YokohamArtLife’
Special Support Project for Artistic Creation"
https://yokohamartlife.yafjp.org/(Japanese only)

18

19

Paying attention to the evaluation environment

Evaluation pitfall ❸Coco thought that it was best to invest in athorough evaluation. She sought out alocal consulting company andcommissioned a third-party evaluation.
Even though they produced asplendid-looking report, she didn’t feelthat it could be of use in her group’s
activities or in deepening broader
understanding of her project.

RE PO RT

When planning an evaluation process, imagine what you want to bringabout through the evaluation and then choose an evaluation method
appropriate to your goals. If you choose an improper evaluation
method, you will not gain meaningful results. Think about “when,”“with whom,” and “what kinds” of dialogue you want to facilitate
through evaluation activities. “When” refers to the schedule of evaluation activities: before, during, or after the project’s implementation?
“With whom” refers to who you want to collect data from: for example,
people implementing the project, participants, or members of the local
community. “What kinds” refers to the objectives of the dialogue; is it,
for example, dialogue in order to improve the project, or to communicate the meaning of the project to a particular group?
People carrying out a project look at it from an implementation
perspective. They might have a strong understanding of its content,
but they do not know how it appears to participants, or how it isperceived by partner organizations or local residents. Even if theimplementers talk about how to develop it, there are often few opportunities for them to discuss how they think and feel about it in depth.
Evaluation is the act of looking at a project you might see on an
everyday basis from someone else’s standpoint to gain a three-dimensional understanding. By engaging in dialogue with various stakeholders, you can learn about how they experience the project and how to20

improve it. This is the principal benefit of evaluation.
Even if you entrust your evaluation to a professional, a one-sided or
one-dimensional evaluation is not evaluation in the true sense. Before
engaging an outside evaluator, consider from what perspectives you
would like them to carry out the evaluation, how you would like tomake use of the evaluation results, and “when, with whom, and whatkinds of dialogue.” Let the evaluator know about these points inadvance. “Third-party evaluation” is a broad term which includes boththe stand-alone method, in which the evaluator observes projectplanning and management from the outside, and the empowerment
method, in which the evaluator provides advice on planning andmanagement. If you want to create opportunities for dialogue after
project implementation, choose the stand-alone type; if you want toencourage dialogue during the implementation of the project, choose
the empowerment type.
In addition, consider the degree of public awareness of the project, as
well as the evaluation goals, project content, and budget. If the significance of the project is widely recognized, for example, improving
project planning and management are important, so obtaining athird-party empowerment-type theory-based evaluation (a method
which evaluates the level of consistency between project goals andcontent) or a management evaluation could be most effective. On theother hand, if the significance of the project is not widely recognized,
publicising the project while working to improve it through a participatory-type outcomes evaluation is recommended. Large-scale projects
can also demonstrate their effectiveness through third-party
stand-alone outcomes evaluation using economic indicators. (See
Chapter 3)

Evaluation suited to the unique nature of cultural projects

Evaluation suited to the unique nature of cultural projects

Point 3

There are numerous evaluation methods, each with its own benefits
and limitations. It is important to ascertain which method is the mostappropriate your project. Whichever method you choose, stay awareof “when, with whom, and what kinds of dialogue” you want tomaintain focus.

21

Evaluation suited to the unique nature of cultural projects

Summary
In Chapter 1, we explored evaluation suited to the uniquenature of cultural projects. Remember to pay attention to
the following three points in order to avoid “evaluation
pitfalls.”① First and most importantly, discuss the public value of
your project within the implementing team. It is essential todiscuss which aspects of your activities include the “valueof socially inclusive arts activities” and to focus on theseparts when carrying out your evaluation.
② Building on this, think about how you can make thisvalue visible. There are methods of expressing the value of
your activities in a three-dimensional way, combining
qualitative data, such as stories from the ground or photographs, with quantitative data. You can also creatively
devise your own indicators instead of relying on existing
metrics.
③ Moreover, when you begin your evaluation, think about
when, with whom, and what kinds of dialogue you want tohold. Evaluation is the practice of looking at activities you
take part in on an everyday basis from someone else’s
standpoint, and discovering areas for improvement
through the process. A deep awareness of the environments and dynamics of evaluation dialogues will lead you
to the most appropriate methods to reach your evaluation
goals.

Taking a peek at the evaluation
process
So far, we have covered some considerations which areimportant keep in mind when engaging in evaluation. How,
then, do you actually evaluate your project? In this chapter,
we will introduce some real-life case studies and examine
the practicalities of evaluation. The results are one
important aspect of evaluation. But the evaluation
planning process, through which you decide how to comeup with the language to accurately evaluate your work,
can teach you how to communicate the value of youractivities to others. Let’s take a look at that process.

As such, when planning an evaluation, always be awarethat socially inclusive cultural projects are value creation
projects.

22

23

Planning an evaluation process leads to new insights

Now I understand common
mistakes and what’s importantto say!

But it feels so difficult...
It’s hard to work up any
enthusiasm about it...

You may understand the importance of evaluation but find it hard towork up any enthusiasm to actually do it. Or you may want to try
evaluation but not know where to start. It is true that tackling evaluation takes time and effort. As we touch on in other chapters, there
is no single foolproof evaluation method, which can make it seem
even more daunting. Once you get started, though, it leads you torethink your activities as you reflect on them. The process of trial
and error as you make decisions on different aspects of the evaluation method generates a lot of growth and learning.

Taking a peek at the evaluation process

Taking a peek at the evaluation process

Key points

Project overview

The evaluation process itself
brings up lots of important things!

What kinds of things?

Lots of interesting questions
and ideas came out of theprocess that I led at Momochi
Culture Centre. Let me tell you
about them!

Case

24

Socially
inclusive
initiatives
began at Fukuoka Prefecture’s
Momochi Culture Centre around2017. Before this, the Centre
was responsible for coordinating
drama workshops for special
needs children and setting up a
Provided by: Fukuoka Prefectural Momochi Culture Centre
older adults theatre group. One
aspect of this work is “Workshops to Experience the Fun ofExpression,” which began in 2018. These drama and danceworkshops are aimed at disabled people in Fukuoka; people withintellectual disabilities, developmental disabilities, and mental
illness have participated in them so far. Fukuoka-based artists
based have been invited to act as facilitators. At the end of theannual series of about ten workshops, there is an opportunity toperform. Participants’ daily lives and the ideas they express invarious forms through the workshops become the raw materials
for a dramatic piece. As a result, the level of participant satisfaction with these workshops is very high.

25

An “evaluation team” was set up for these workshops, with Professor Nagatsu’s research group from Kyushu University’s Faculty of
Design, as the result of a proposal by Momochi Culture Centre staff,
who understood the importance of evaluation and validation fromthe outset of the project. It was decided that the validation teamwould observe workshops from the outside and carry out a qualitative analysis through questionnaires and interviews.

［Workshops to Experience the Fun of Expression］
Fukuoka Prefectural Momochi Culture Centre
Designated administrator: Momochi Palace Network
JTB, Fabils Co., Ltd., Kyushu Stage Art Manage Association,
Art Management Center Fukuoka

Facilitators:

Collaboration:

Nobuyuki Gomi,
Kyoko Koga

Special Olympics
Nippon Fukuoka

Assistant:
Sae Tamura

26

Validation team:

Professor Nagatsu’ sresearch group,
Faculty of Design,
Kyushu University“We want to communicate the project’ s value more
widely!”In 2019, the staff of Momochi Culture Centre told Professor Nagatsu’s research group they wanted to communicate the value of the
project to a wider, external audience. But they had run up against
some uncertainty about how to evaluate the project since there werevarious stakeholders involved, each with their own expectations of
the project, including participants, their families, staff members ofcultural facilities, artist facilitators, and many others.

Taking a peek at the evaluation process

Taking a peek at the evaluation process

What is evaluation actually like?

Beginning by listening to the people involved
We therefore decided to begin by listening carefully to what each
person hoped to get out of the workshops. This idea is based on an
approach called participatory evaluation in which everyone involved
participates in the evaluation process: not only those offering theactivities, but also those on the receiving end.
We initially conducted individual interviews with the people involved,
using this approach. We applied simple coding based on what
emerged from these interviews, seeking to understand both sharedand individual expectations of the workshops. After the interviews,
we held a meeting in which everyone involved discussed workshop
goals.

Participants27

Two shared goals emerged as a result. One was “disabled people
and the people supporting them create new forms of expression andrelationships through culture and the arts.” The other was “cultural
facilities become places where people in diverse situations can playand interact through theatre.”
In addition, artists, participants, family members, and staff membersof cultural institutions each expressed the value of the workshops tothem. Through this process, everyone was able to articulate theunique characteristics of the workshops. They were not necessarily
seeking to create novel and innovative works, nor to do somethingfor disabled people. It became evident that what was important was
the emergence of new relationships and the creation of a place inwhich the people involved could interact with each other.

Interviews
When we asked everyone for their views, wefound that many people thought it was important tohave the space and freedom to tell each otherwhat they think, leading to greater self-expression.

Defining goals
At these workshops, it is important that

expression develops out of the views of people
from diverse backgrounds!

Articulating what is happening
In observing the workshops, it is interesting to

VO

ICE

S

Taking a peek at the evaluation process

Taking a peek at the evaluation process

Neither creating interesting pieces,
nor “doing something for disabled people”

see how the artists respond to the “voices” ofthe participants from moment to moment,
spontaneously co-creating new meaning.

Putting “participatory evaluation” to use
In participatory evaluation, the values described above are understood as goals the team goes on to create evaluation scales, and thescales are used to measure progress. With the Momochi Culture
Centre project, though, when it came time to measure progress, we
paused because the participants, who were at the heart of the
project, had not been included in the discussions yet. We considered
how to access the opinions of the disabled participants and decided
to devote one series of workshops entirely to the verification of thegoals themselves. By picking up on the minute incidents which
occurred during the workshops using qualitative research methods,
we tried to verbalize what was happening in their workshops. Afterthe workshop period was over, we held another discussion where we
presented the results of the research group’s observations and wereable to rework our goals.

28

Reviewing and organizing goals
If we say “the views of people from diverse

backgrounds,” it sounds as if these have to
be expressed in words, doesn’t it?

Perhaps more important than an innovative
expression is the birth of a “shared creative
space,” where “participants can relate toeach other as protagonists in creation!”

Proposing evaluation scales and data collection
Let’s explore “the degree to which theparticipants and artists feel that they are
relating to each other as protagonists increation” using questionnaires and interviews.

To be continued...

29

What kinds of things become visible when we implement evaluation
like this? Of course, the results of evaluation are important, but what
the evaluation itself brings is also significant. The following are a few
such examples.

From “evaluators” to facilitators who enable everyone to join in
In socially inclusive arts projects like the one in this case study, there
are no predetermined evaluation standards. For this reason, we
decided to listen carefully to each of the participants and analyse
what we heard, leading to a process in which everyone createdevaluation standards together. This might have led to an increased
awareness that participants themselves were not the “subjects” ofevaluation, but rather that they and other stakeholders were taking
part in a process to move the project forward together.
When this happened, it was not helpful for us, as researchersaccustomed to acting as expert evaluators, to assess the project
according to a single set of beliefs. Rather, our role was to act as
facilitators in reconciling the different views expressed by thepeople involved.

Strengthening communication between stakeholders
The participatory development of evaluation standards led todialogue between people who generally did not communicate witheach other outside of the workshops. It created a space for them to30

communicate about their thoughts and feelings. Artists, participants,family members, project staff, and others come from different perspectives, but relationships in which they could hold frank discussions ona shared topic were born. This led to smoother implementation of theworkshops and perhaps to a change in how the family members
perceived the project.

Taking a peek at the evaluation process

Taking a peek at the evaluation process

Trying it out

Evaluation became a place in which the seeds of new
programmes were planted

An even more interesting aspect of this space was that the participation of the facilitators in the evaluation process brought about changes to the programme itself. One of the facilitators who took part in
the evaluation discussions learned about the expectations of various
stakeholders and in particular heard disabled people and theirfamilies talk about their various hopes for the workshops in relation
to their everyday lives. This facilitator later proposed “family” as thetheme of this series of workshops, and various programmes onfamily were later held. What began as a discussion for the purposesof evaluation became an opportunity to understand the audienceand assess needs in order to build programmes suited to the service
users.

31

Drawing out and weaving together values
At the time of this writing in 2020, evaluation of the project is ongoing.
We are developing evaluation scales based on the goals, designingquestionnaires, and collecting responses from the artists and participants at each workshop. We are also planning to hold a meeting forfamily members to reflect on the programme. We will take great care
as we craft a narrative based on the results to communicate thevalue of these workshops more widely.
From time to time, we will probably have to return to goal-setting,and as the activities take place, new visions will probably become
clear. A place is coming into being in which everyone can feel thatthese are simply opportunities to think together once again.

Key points of Momochi Culture Centre’ s evaluation
Evaluation design
The aim was to make visible the “value of socially inclusive arts activities”(Chapter 1, pages 14 – 16) as expressed in the thoughts and feelings abouta project developed by various stakeholders.
Consideration of indicators
Through interviews, discussions, and observation of the workshops in action,
evaluators collected and applied data from the various stakeholders toarticulate the value of the workshops.
Data collection

Taking a peek at the evaluation process

Taking a peek at the evaluation process

Evaluation has no end:

While keeping records of progress including video footage, evaluators
designed questionnaires based on the indicators, allowing them to gain
understanding of what was happening at the workshops.
Use of the results
In addition to disseminating evaluation results to society at large, the project
team noticed the importance of paying attention to interesting interactions
which happened during the evaluation process.
A key learning from the evaluation at Momochi Culture Centre is that the“evaluation cycle” does not necessarily proceed in a set order. People who
evaluate socially inclusive arts activities are required to think in a somewhat
different way than in other evaluations or assessment. There are also
elements which the project staff can do themselves in evaluating a project.
For example:❶ Have a discussion with everyone involved to start developing project goals.

❷ Try to develop ways to investigate whether or not these goals have been achieved.

❸ Have several people keep progress records throughout the project.

❹ Hold a debriefing meeting when the project is completed.

32

33

The results of an evaluation are of course important, butthere are also many things which emerge through theplanning and implementation of an evaluation. In thischapter, we introduced the process of initiating evaluation and all that it involves.
Setting your own goals, thinking about validation
methods, reflecting on the project at the end, andlinking results to the next set of goals... moving forwardwith the project while staying aware of evaluation
allows you to notice unexpected values you may nothave observed otherwise. It can also empower thepeople involved; enable the crafting of a new, common
language; and create ripple effects for the programme
itself. In other words, it can be said that the development of a space for evaluation is in itself a creative
process.

Four case studies

カスタマイズのプロセス

Taking a peek at the evaluation process

Summary

In this chapter, we present case studies
based on interviews with members fromorganisations implementing social inclusion
arts projects, or from funding bodies using

distinctive types of evaluation. They shared
about the goals and circumstances prompting
them to conduct evaluations, as well as theirevaluation procedures and results. Use thesecase studies as starting points to think about
the most appropriate evaluation methods andprocesses for your activities.

34

35

Key points

?

?

?

Here, we introduce four case studies shared by people who have
implemented evaluations of their social inclusion arts projects.
Based on these cases, try to identify the most appropriate evaluation methods for your activities as you plan your own process.

Experts

We don’t have any
evaluation experts
we can rely on.

Even if we write a report,
it won’t convince the government,
will it?

We are all colleagues, so we can readily accept each other’ s
opinions! Peer review evaluation: A method that does not needexperts

行政

REPORT

Your group can learn to carry out evaluations independently,
with advice from experts. Empowerment type project
improvement evaluation

RE
PO
RT
00

P.38

Implementing organisation: Toride Art Project

ZZZ…

00

００ 0 0t
Go ve rnm en

Four case studies

Four case studies

As we have seen, tackling evaluation is an ongoing process filledwith twists and turns, trial and error.

I feel as though I’ve learned a lot and understand evaluationin detail now, but...
I don’t know how to assess our activities.

P.44

Grants secretariat: Arts Commission Yokohama

There are various
evaluation methods,
you know!

OH!!

evaluation

evaluation

evaluation

evaluation

evaluation

Mmm... We don’t really
understand the differrencesin evaluation methods orwhat they involve. How dopeople choose a method?

We want to illustrate the impact and social role of music and art
in partnership with researchers! Creating unique indicators

evaluation

Implementing organisation: Tokyo Bunka Kaikan, Tokyo Metropolitan
Foundation for History and Culture
A strategic approach to government!
SROI evaluation, converting the project’ s social value intomonetary value

Why don’t we try asking people
who have done evaluations
about why they chose aparticular method?

P.50

P.56

Implementing organisation: Kani City Cultural Creation Center ala
List of case studies

P.62

There is a list showing all fourcase studies at the end, incase you’d like to look at thatfirst and read the ones most
relevant to you.

36

37

Please tell us what prompted you to carry out anPeer review evaluation● Implementing organisations partner with organizations or colleagues(peers) in the same field to evaluate each other’s projects

First, we were only able to show the external evaluators one
aspect of our project. We worried about the extent to which weshould share about the long-running nature of our work as
material for the evaluation. Furthermore, we were not able toshow evaluators the truly interesting moments of our art project,such as when the local men sit on a bench and say interesting
things.

Evaluation goals

Finding ways to carry out projects using methods other than
third-party evaluation

What are the mainthings evaluated?

The project as a whole (evaluation is set up by project)

External experts

Unnecessary

Who carries out
the evaluation?

Reciprocal evaluation by others in the field (peers)

Implementing
organisation

Toride Art Project

Name of the project
evaluated

All of the projects undertaken by Toride Art Project

Project outline

Toride Art Project (TAP) had the opportunity to be evaluated byexternal experts in 2013. Afterwards, though, TAP members felt
frustration and regret about various aspects of the evaluation.

After carrying out festival-style and year-round projects,
Toride Art Project decided to shift the focus of its activities,
aiming to become a full-scale arts centre. Participants inthe “& Geidai” art-management human resource development program run by Tokyo University of the Arts’ Graduate
School of Global Arts worked on this peer evaluation as oneof their practical assignments.
Referencematerials

Second, the evaluation ended up simply encouraging andaffirming the project, perhaps because we gave the external
evaluators insufficient information or did not convey its
complexities well.

﹈Peer review evaluation﹈
Peer review evaluation

Because peer review
evaluation is done by peers
doing similar work,
evaluation and criticism canbe shared constructively.

evaluation, and why you chose this evaluation method.

﹇
Four case studies﹇
Four case studies

Toride Art Project and Outenin Temple

After this experience, we learned about the concept of peer
evaluation in 2015. We were interested in this approach, which
was different from anything we had done, and decided to try itout. The advantage of peer evaluation is that assessment andcriticism is easier to accept because it is given by people in yourfield. You have the opportunity to be confronted with your own
issues through the mirror of your peer. At the same time, you learn
that your counterpart is also worrying about the same kinds ofissues.

p. 80, ①④⑤

Photo provided by: Toride Art Project38

39

In TAP’s case:1) Understanding projectstakeholders
2) Involving diverse community
members in setting up the NPO
3) Creating "a space"

Selected learnings from the peer evaluation process

During implementation

Share materials (materials about the organisation’s management and operations, questionnaire, etc.)

Observe the peer’s project at work (learn about external-facing activities such as events, hubs)
Site visit to the peer’s office (learn about management and operations, etc.)
After the visits, the TAP
members review the meeting.

●Share feedback and impressions

Presentation at an “& G” study group meeting

Later

Review and hone the organisation’s internal communications
methods

What were the outcomes?

Immediately after
implementation

Summarize findings in a report

After deciding on Outenin Temple as our peer, we then set the
topics that we wanted to examine, shared documents from ouroperational offices with them, and sent a questionnaire tothem. We visited Outenin Temple and debriefed the meeting
after coming home. Through discussions with the members,
we organised their opinions (see chart below) and took thediscussion deeper for the “& G” study group.

What was happening at Outenin Temple

Insights

At Outenin Temple, the office had planned
and run virtually all aspects of project
management, but recently they hadradically changed the role of the office tobring about a situation in which “Quality＜ Taking on a challenge”.

·The idea that “Quality ＜ Taking on achallenge” is new and refreshing·The dilemma that the same people
always end up taking charge·It’s great that an office which had
always been in charge of planninghas taken a step back.

﹈Peer review evaluation

Set the aspects to beinvestigated by the peer review

Accept the offer

We first looked for an organisation similar to TAP, one thatworked based on the “hub model” and had been active since
about 2000. This took some time, because there were no Tokyo
organisations using the hub model and active for this length oftime. Also, we couldn’t find suitable dates with the firstorganisation we considered, and it was hard to find time for
evaluation while still running our project hub.

﹇Four case studies

Make an offer

What did you actually do?

Search for a partner for the peer
evaluation (a project that seems
similar to yours)

Update the website to improve
dissemination of project information

40

Outenin Temple(peer)●﹈
Peer review evaluation

Before project implementation﹇
Four case studies

TAP(organisation implementing the project)

Thinking about the commonalities between TAP and Outenin
Temple was a fresh approach. After undertaking the peer
evaluation with Outenin Temple, we gained a renewed
understanding during internal discussions of the obvious yet
fundamental point that while the value of the Art Project cannot
be determined, the hustle and bustle generated has value. In
addition, within our workplace, we were able to confirm what
each of the staff members of TAP thought and felt. For
example, the working conditions at the Art Project are not very
good. We were able to name the reasons why each of us still
continue to be passionate about and involved with TAP, andwhat we personally consider most valuable about the activities.

41

AyakoOmuka

How are you making use of the evaluation results?

Since TAP runs each of its four projects at different hubs, staff
rarely have the chance to meet each other face-to-face. Before
the peer evaluation, project staff were hesitant to ask for input
from all of their extremely busy colleagues, feeling that theyshould do a certain amount of planning on their own, and let themknow by email. However, after the peer review evaluation, theatmosphere changed; taking the time to consult with each otheron certain project decisions became more important. During thepeer evaluation, staff were able to see that other people often
have answers to contribute, and that talking to others in their field
can lead to essential insights.
Is there anything we should be careful about when doinga peer review evaluation?
I think a peer evaluation is meaningful for any group, however
small. Simply learning about one’s counterpart, without making a
big deal about the fact that it’s an evaluation, leads to many
discoveries. However, it’s hard to conduct an evaluation whilerunning a project, so it’s probably a good idea to include peer
evaluation as one of the core aspects of your project. It is also
important to be aware that in peer evaluation, a lot of time isneeded to select a peer and arrange a schedule. The internal
review is the most important part of peer evaluation, so be sure toallow ample time for this, too.

42

Sumiko Kumakura’s experiences and issues on the ground
When I studied Drucker’s method of non-profit management, I
was taught: “If you evaluate yourselves, it turns into awkward
nitpicking.” I had my own experience of trying this and failing, andso I came to feel that it was difficult to introduce any evaluation
unless those working on the ground welcomed it. Also, while theart projects themselves continued to flourish, the fact that the
young project staff members had few opportunities to meetothers in the same field, the way that staff members of public
cultural facilities can do at trainings, was another issue.
The power of the arts, and the problems with logic models
If you’re going to work on a cultural project, you need to think
about what the arts are. The arts generate unexpected results
through previously unknown potential or the power to break freeof existing systems and concepts. It’s as if you asked for curryand rice but were served fried noodles. It’s fine to use a logic
model, but if meeting targets becomes the only goal, this is a caseof putting the cart before the horse. The power of the arts is lost,
and the implementation of the project, which works through thearts, will be diminished.
Distinctive features of peer evaluation
Peer evaluation provides an opportunity to encounter others in
the field and to capture terms and values which cannot be understood with a logic model, and which can slip through the cracks
otherwise. The method highlights the fact that looking only atthings which yield results is not evaluation.
3 key points

Photo provided by: Toride Art Project

Saya
Makihara﹈
Peer review evaluation﹈
Peer review evaluation

We made use of the peer evaluation to improve our operations
and promote internal communications within our workplace.
There were astonishing results, particularly in terms of internal
communications.

SumikoKumakura

We talked with Sumiko
Kumakura and Saya Makihara,
who are in charge of “& G” at
Tokyo University of the Arts!

﹇
Four case studies﹇
Four case studies

Yasue
HabaraWe talked with
Yasue Habara and Ayako Omuka
of TAP!

1. Your peers have many useful insights that can support
your project.
2. Peer evaluation is useful for improving internal communications and management.
3. The staff team should review the project to reconsider
assumptions.

43

Empowerment-type project improvement evaluation● The grants secretariat gives some workshops to encourage
self-evaluation by implementing organisations. In addition,
selection panel members give feedback before, during, andafter implementation.

Evaluation goals

Support implementing organisations projects to facilitate
improvement, while also understanding the importance ofevaluation of their various activities.

What are the mainthings evaluated?

TheoryExternal experts

Necessary

Who carried out
the evaluation?

Implementing organisations, experts in the arts and community building

Grants secretariat

Yokohama Arts Foundation, administered by the Arts Commission Yokohama (ACY)Name of the project
evaluated
Project outline

Ever since ACY began grant programmes in 2008, we have been
involved in a process of trial and error concerning not only evaluationof implementing organisations and of artists receiving funding, but of
the grant programme itself. In the 2010–2011 fiscal year, we held
intensive study groups and workshops on evaluation, and produced
a report on the nature of evaluation. At the time, however, we simply
created a report without applying it. Thinking about it now, we didn’t
really have a perspective on how to improve the projects; we had astronger focus on appraisal and checks.
It was during Creative Inclusion (a project to support artists working
in socially inclusive activities) in 2017–2018 that the importance ofevaluation was really brought home to us. At the time, we decided to
do evaluation for ourselves, rather than for someone else, andcarried out an empowerment-type project evaluation. We used alogic model and other tools from the design stage of the grant
programme, discussing with evaluation experts how to put together
a programme which would effectively support the selected artists
and organisations. We asked the selection panel members not only
to select the projects, but also to give each organisation advice
before, during, and after implementation. This led to visible changes
in artists and organizations
The next task was to ensure that the impacts were disseminatedthroughout society. As a result, in 2020, we designed YokohamArtLife,an open recruitment programme that pays attention to outcomesfrom the outset. We co-created evaluation indicators with the artists
and organisations selected as one aspect of our support to them.

YokohamArtLife held in the 2019-2020 fiscal year
A two-year grant programme to support projects encouraging people who have not had access to the arts and culture
to participate in such activities.
Referencematerials

44

evaluation, and why you chose this evaluation method.

﹈
Empowerment-type project improvement evaluation﹈
Empowerment-type project improvement evaluation

Thinking about the design of grants
programmes with evaluation experts
and supporting organisations leads
to a better project overall.

Please tell us what prompted you to carry out an﹇
Four case studies﹇
Four case studies

Arts Commission Yokohama

p. 80, ⑥ - ⑧

45

Participate inworkshops, create
indicators

Hold mid-term reporting session
Report on projectprogress

Hold year-end reporting session

Later

Ongoing project
development(planned)●

Submit a final project
report at the end of
the fiscal year

Present reports

46[Selection panel members]
Advise implementing
organisations[Selection panel members]
Advise implementing
organisations

Are selection panel members apart of the process throughout?

Immediately after
implementation

Compile a project
report document basedon the reports
submitted by eachorganisation[Evaluation experts]
Facilitate workshops

However, the examples presented for the common indicators in
the first year were too strongly oriented towards local government
indicators or organisation-specific matters, (e.g., level of satisfaction
with Yokohama City, isolation and challenges in people’s lives,)
rather than artistic and cultural experiences, so the participatingorganisations first reacted with comments such as “We’re notcarrying out our activities for goals like these.” As a result, theindicators ended up being typical ones, such as “Frequency ofartistic and cultural experiences.”
So, in the second year, we added psychological indicators (e.g.,
inspiration, self-esteem, connection with society) reflecting
participants’ experiences of the project to the list of “common
indicators.” This time, everyone expressed interest and was eager
to participate. In addition, at the same time as making a simple
logic model for the “individual indicators,” we worked to find theoverlaps between what each group was doing and what wasneeded in society, using this to inform our indicators. This work
also ended up serving as a theory-based evaluation, improving
the quality of their activities.

We ask the selection panel members not only to select theartists and organisations implementing the projects, but also to
participate in mid-term and year-end reporting sessions to review
each organisation’s activities.

﹈
Empowerment-type project improvement evaluation

Announce projects[Selection panel members]
Advise implementing organisations

One or two representatives from each implementing organisation
came together and, along with our secretariat, created “common
indicators” for use across all organisations, and “unique
indicators” to guide the self-evaluation of each organisation, withadvice from experts.

﹇
Four case studies

Hold selection panel meeting

What did you do at the evaluation workshops?

Develop and publicise
grant application
guidelines

On the application form,
require the descriptionof project content aswell as evaluation
methods that pay
attention to outcomes.

Support the design ofthe grant programme●

Through a logic model,
design a grants
programme which pays
attention to outcomes

Experts

Hold 3 to 4 evaluation workshops
During implementation﹈
Empowerment-type project improvement evaluation

Before project implementation﹇
Four case studies

Implementing
organisation

Grants secretariat

As such, the activities implemented continue to improve. Our
grants scheme pays attention to the outcomes of the scheme
itself and is designed to encourage the growth of artists andorganisations.

47

Eisuke
Sugizaki

What did you achieve?

How are you making use of the evaluation results?
At the end of the project cycle, we create and publish a report
evaluating ACY’s grant programme and each funded project. In
preparing the report, we pay careful attention to who the subject is.
We also put a lot of thought into the effective use of photos tocreatively demonstrate the impacts of arts activities.

It helps if you think that the issues are not in society, but instead arein the realm of the arts. I don’t think we should evaluate whether weare solving the problems of society or not; if we believe in the powerof art, we just need to look at how widely the impact of the arts isbeing disseminated in society. We need to assess if we arecommunicating the value of our activities to broader society, and if
we are communicating effectively with each other. This is something
we need to resolve first.
The responsibilities of funders
If funders simply hand over the money and abdicate all further
responsibility, this will not lead to an evaluation cycle, however hard
those involved think of logic models.
Funders must clearly explain why they are offering the grants toimplementing organisations. It is important to explain their own goals
and understandings of what the funding means, and what it is for, sothat organisations can identify how their own goals and meaning
connect and pursue these connections: this is what we expect fromthem. I think this is the start of evaluation.

By carrying out robust evaluation of the grants programme andlaying out the issues methodically, we have become able to makeconcrete proposals on the kind of programme we would like toimplement next when Yokohama City government or others ask us
for advice. In the end, evaluation also functions as advocacy.
3 key points

48﹈
Empowerment-type project improvement evaluation﹈
Empowerment-type project improvement evaluation

In the second year, each organisation became able to evaluate thenew values generated by the overlap of “arts venues and spaces”
and “community venues and spaces.” There was an increase in thenumber of concrete activities leading to project improvement, suchas the creation of questionnaires focusing on this overlap or the
sampling of stories from journals kept by staff members. At the
mid-term reporting session in the second year, in particular, we could
see that each organisation was using proactive and ingenious
methods to disseminate the benefits of its activities more broadly
and was conscious of the question of sustainability of its initiative.
So we felt confident that the evaluation process was yielding results.

Advice for people struggling with evaluation﹇
Four case studies﹇
Four case studies

We talked with
Eisuke Sugizaki of ACY!

1. By figuring out their own indicators, implementing
organisations can make the value of these projects
visible and work to improve them.
2. When the grants secretariat clarifies the goals andpurpose of the funding and communicates thisinformation to organisations, the evaluation cycle canbegin to function effectively.
3. The grants secretariat is then able to make concrete
proposals to local government about grant
programmes and policies.

49

Please tell us what prompted you to carry out anevaluation, and why you chose this evaluation method.

Creating unique indicators

Evaluation goals

Clarify the project’ s meaning and efficacy from an objective
standpoint, leading to project improvement and programme
development

What are the mainthings evaluated?

Outcomes

External experts

Necessary

Who carried out
the evaluation?

Researchers in the psychology of aging and in the sociology
of artOrganisation
implementing the
project

Tokyo Bunka Kaikan, Tokyo Metropolitan Foundation for
History and Culture

Name of the
project evaluatedInclusive Arts Initiative held in 2018 – 2020(music workshops for senior citizens, lectures and training
for the art activities leading to social inclusion

Project outline

Music workshops with an emphasis on creativity, cooperation, and inclusivity were held at Tokyo Bunka Kaikan, cultural facilities, facilities for senior citizens, and social welfare
facilities in Tokyo as Inclusive Arts Initiative. Training
sessions for artists and lectures about social issues werealso held.
Referencematerials

50

We have been working on socially inclusive projects at Tokyo Bunka
Kaikan since 2016. As society’s demands of the arts and culture
and of cultural facilities and organisations increase, under thecatchphrase “arts and culture which contribute to society” we felt aneed to show the social role of the arts objectively, through thevalidation of case studies from our work. We seek to answer thequestion of what changes music, and the arts in general, bring about
in people and in society. Since 2018, we have asked Dr. Nahoko
Kusaka, a researcher in the psychology of aging at Doshisha
Women’s College of Liberal Arts, to carry out third-party evaluationand validate our activities. ＞ information on evaluation, p. 50 – 51
The biggest reason why we asked Dr. Kusaka was that she is apractitioner as well as a researcher. We first met her during onsite
observations and then read her workshop verification for seniorcitizens given by another organisation. Since then we had wanted towork with her.

﹈Creating unique indicators﹈
Creating unique indicators

By validating and analysing themeaning and outcomes of yourown activities with researchers,
you will be able to review youractivities and promote related
information more effectively.

ents
vem
hie
Ac

● Researchers and the implementing organisation create
indicators to measure the project’s uniqueness● Through fieldwork, the value of the activities is articulated.

﹇
Four case studies﹇
Four case studies

Tokyo Bunka Kaikan

We are currently conducting a verification of social issues such as
extending healthy life expectancy and improving QOL from theperspective of exploring the power of music, using an action
research* approach based on our original music workshops forsenior citizens. In concrete terms, we worked together with auniversity on an objective verification, incorporating knowledge fromthe psychology of aging to analyse the changes which occurred inworkshop participants during or shortly before or after theworkshops by looking at the reactions of or changes in the research
participants.

* Action research: A research method which aims to find solutions to issues. Those
directly concerned think with researchers about solutions to the issues, validate thesesolutions, and resolve the issues based on the validation results, while also looking atways to improve the solutions.

p. 80 – 81, ⑨ - ⑯

51

Define research plans, methods, and issues. Identify research questions
from preliminary researchGain consent from research
participants
Coordinate between researchers,
others involved, and artists

During implementation

Hold workshop(s)Immediately after
implementation

Reflection

Review research ethics(researcher)

Conduct pre- and post-project interviews
Collection of video and audio data throughbehavioural observation (including field notes)＞ information on evaluation, p. 50 – 51

For example, in 2019, we carried out validation on the topic of“creating a framework which will make people who do not havea habit of engaging with cultural activities want to participate”.
We analysed the workshop programme, and validated whether
or not participants’ moods, such as “relaxed”, “excited”,“comfortable”, etc. changed before and after the workshop,
and whether or not differences could be seen in these moodchanges depending on participants’ degree of health, feelingsof unease, or experience of participating in musical activities.
In 2020, based on these results, we focused on the perspectiveof the programme to group satisfaction and happiness,
modifying the design to make it easier to feel empathy,
achievement, and gratitude in the context of relationships withothers.

﹈Creating unique indicators

Recruit participants, arrange
performers, venue, etc.

We decide on concrete topics for each fiscal year. Then we
explore the outline of the research based on our hypotheses
and, to validate these, measure subjects’ behaviour, andconduct questionnaires and interviews. Using diverse researchmethods and psychological theories, we are able to convert
the changes in the participants into indicators from a variety ofviewpoints.

﹇
Four case studies﹈
Creating unique indicators

After selection of researcher and coordination, share fields and project content● What kind of research and validation
did you carry out?

Confirm a field，pre-coordinationand liaison

Conduct pre- and post-project questionnaires●

Publish report or hold reporting session

Later

PR within and outside theorganisation
Carry out ongoing research andvalidation

What did you achieve?

AnalyseCarry out project improvement

52

Researcher

Decide on fields, projects, or
topics for research andvalidation

Before project implementation﹇
Four case studies

Tokyo Bunka Kaikan(implementing organisation)

Taking the example above, it became clear that, for adults
participating in a workshop for the first time, a structured
programme with a clear goal – what they will do at the workshop,
and what they will become able to do – is effective in increasing
participants' sense of security.
Also, during the workshop reflection, artists sometimes says
“I was wondering about the meaning of this action by thatparticipant”. Then the researcher may response with her analysis
from the lens of behavioural psychology . This can be a trigger for
artists to act more consciously and become more aware ofeffective actions (interventions). This allows us to improve theworkshop programme as a result.

53

Yukiyo Sugiyama

Did setting up workshops involving researchers create
any difficulties for you?

We are using various means to create opportunities for communicationfrom the pre-implementation phase onwards. We write down thecontent to be researched and our to-dos on the day in detail andconfirm these in advance. We pay careful attention to ensure that the
researchers do not create unnecessary interventions. We also ensurethat the artists and researchers can communicate as non-hierarchicallyas possible.
How are you making use of the evaluation results?

In 2019, we asked Mia Nakamura, a researcher in the sociology ofart from Kyushu University, to conduct fieldwork and interviews
with specialist agencies and practitioners. Our goals were to clarify
the strengths and weaknesses of our inclusive projects, and their,
implementation structures and to verbalise the value (outcomes
and impact) of the activities in order to use these as evidence forour projects.
As a result, we were able to verbalise the value of the workshops,
which we had only previously captured in participants’ accounts of
their experiences, as well as the role of the workshop leaders(specialists who can communicate the joy and fun of music to awide variety of different groups from new angles) whom we aretraining at Tokyo Bunka Kaikan, and the importance of andchallenges with the training programme.

﹈
Creating unique indicators﹈
Creating unique indicators

Some unforeseen trouble occurred on the first day of a workshop at
a care home for senior citizens, due to a misunderstanding during
our prior confirmation of the research process. Liaison andcommunication between those involved is essential for action
research; since we are carrying out our research at the same time asholding the workshop, it is extremely important to build good
relationships and trust between the artists, workshop participants,
researchers, and facility personnel and create an environment inwhich everyone can feel comfortable and act in their usual ways.

Evaluation from a new perspective﹇
Four case studies﹇
Four case studies

We talked with Yukiyo Sugiyama
of Tokyo Bunka Kaikan!

The results of this research are reflected in the “Guidebook to the
Art Activities leading to Social Inclusion” published in March 2020
(English version published in 2020), and are contributing not only to
advocacy but also to other organisations undertaking similar
cultural projects or willing to start inclusive programmes.

It is important that evaluation and validation illustrate the value andimpact of our work based on the project outcomes, from a third-party
perspective. Internally, we make use of the those reports evaluation to
improve projects and identify our challenges in subsequent years. We
also share our learning in annual reports and at international
conferences.

We hope that clearly stating, disseminating, and sharing the outcomesand knowledge obtained from our activities in these ways will help tostimulate activities nationwide.

54©Mino Inoue

3 key points

In addition, we hold reporting sessions and forums to share thevalidation results and outcomes with other cultural facilities or arts
personnel. By doing so, these can serve as a reference when otherorganisations or regions hold similar projects or create frameworks.

©Jouji Suzuki

1. With experts, create indicators to evaluate the uniquecharacter of your project.
2. It is important to have a good liaison and communication between everyone involved, including experts,
artists, participants, and staff.
3. You will be able to gain a new understanding of yourown activities from an objective standpoint.

55

Please tell us what prompted you to carry out anevaluation, and why you chose this evaluation method

SROI* evaluation● Evaluation experts validate the project and calculate themonetary value of the changes it has brought about.

Evaluation goals

By calculating the monetary value of social value, clarify themeaning and outcomes of the project, and fulfil the duty ofaccountability

What are the mainthings evaluated?

Outcomes

External experts

Necessary

Who carried out
the evaluation?

Experts in SROI evaluation

Organisation
implementing the
project

Kani City Cultural Creation Center ala, Kani City Cultural Arts
Promotion Foundation (public interest incorporated foundation)

Name of the
project evaluated

A project utilizing the creativity of culture and the arts carriedout in Kani City (project to develop communicative ability,
2017 fiscal year)

Project outline

Workshops held for several elementary and junior highschools in Kani City, utilizing drama and dance methods withthe aim of improving children and students’ ability to understand others (accept their feelings) and to speak.

However, we feel that what is important is not calculating themonetary value in itself, but rather analysing why changes were orwere not brought about by the project, feeding the results back intoa logic model, and making use of them to improve the project.

* SROI(social return on investment ): a method of calculating how much of a social
dividend was obtained for the money invested by converting the social value createdby an activity into a monetary value.
Referencematerials

56

We started to think, therefore, that we needed evidence that was
broader than just outputs. Without properly picking up the changes
generated by the project and quantifying them by comparing themwith other statistical data, it is impossible to create objective
evidence. We needed to tell local government (the finance division)and politicians about how the participants had changed using
objective evidence, by quantifying the changes brought about by
culture and converting that value into monetary value, and so we
decided to calculate SROI.

﹈SROI evaluation﹈
SROI evaluation

We make use of this to demonstrate
the monetary value of the changes
brought about by a project, and tocommunicate the project’s necessity
to government (the finance division)and politicians.

Our previous evaluations of cultural projects only covered outputs,
such as audience figures or income generated. We were unable toexpress important changes, such as how participants began to smileor became friendly with each other, using objective indexes. This
made it difficult for us to convey the importance of the projects tolocal government.

﹇
Four case studies﹇
Four case studies

Kani City Cultural Creation Center ala

A photo of a theater expression workshop
in Gifu Prefectural East

p.81, ⑰ - ⑲

57

Schools

Kani City(funder)Current status survey andpreliminary hearings regarding
the implementing company andthe implementing project

Collaborate withinvestigation andinterviews① Stakeholder analysis
Investigate the current
situation at the target
schools/ classes, and carryout pre-project interviews

Collaborate with investigation and interviews② Creation of a logic model③ Creation of an impact map+ design of evaluation
indicators

Duringimplementation

Distribute and collect pre-project questionnaire
Hold workshops
Distribute and collect post-project questionnaire

Immediately after
implementation

Hold post-project interviews

Cooperate in interviews

Analyse measurements andcalculate SROI
Submit report

Report to local
government

Later

Implement project
improvement
Carry out ongoing investigation and validation

58② Creation of a logic modelThis clearly states the inputs (resources invested), activities,
outputs, and outcomes of the project or policy, and connects
them in causal relationships.
In this project, we ordered the elements as follows to show how
they were related, working back from the final outcome of“children’s healthy development” to the inputs.

﹈
SROI evaluation

Collaborate with investigation and interviews

What kind of investigation and validation did you carry out?

Investigate the background of projects
utilising the creativity of culture and thearts carried out in Kani City, anddesign pre-project interviews① Stakeholder analysis
Stakeholder analysis is one of the most important components ofan SROI evaluation. A comprehensive sample of stakeholders isanalysed, and the findings are verbalised and organised. Basedon this analysis, we put into words “what kinds of effects there
are on which people”, and create a logic model and impact map
to meaningfully connect these changes.

﹇
Four case studies

ala (organisation
implementing the project)●﹈
SROI evaluation

Before project implementation﹇
Four case studies

Experts in
SROI evaluation*Excerpted from the logic model① Inputs②Activities③Outputs

Holding workshops

Number of workshops held,
participant numbers(resources invested)

People, materials,
and funds④Outcomes
Initial: Children
experience forthemselves the fun ofcollaborating with others

Mid-term: Children start
to relate to their friends
proactively

Final: Formation ofrelationships of trust
between children
Final: Children’s healthy
development②Creation of an impact map
In order to calculate SROI, we organised the inputs, outputs, andoutcomes for each stakeholder, and stated clear evaluationindexes for each item. In this project, we made items such as“Number of instances of self-expression”, “Interest in expression
by others”, and “Degree of trust of adults” into indicators for thechildren, and “Degree of interest in/ understanding of children/
students” into an indicator for teachers. We validated thechanges in these before and after the workshops throughquestionnaires and interviews.

59What did you learn from the results of the investigation?

Is it hard to take on an SROI evaluation?
Since SROI requires expert knowledge in the conversion of a
project’s social value into monetary value, we could say that it isrelatively difficult. It can be used strategically as a means of
persuading local governments and politicians of the necessity of a
project, but there is always the danger that figures can take on a lifeof their own, so it is important to understand the distinctive
characteristics of SROI before using it.

Currently, the importance of evaluating impacts themselves is not
limited to the field of the arts and culture, but is something sharedwith other fields. Even if you do not use SROI, I think that it would

60

Kisei Ei

How are you making use of the evaluation results?
They are not only being used for advocacy towards the local
government and city council, but also leading to the continuationand development of the project. For example, ala has become a hubconnecting the prefectural Board of Education and the Bungakuza
theatre company. In the 2018 fiscal year, the two signed apartnership agreement, and drama workshops are being held on an
ongoing basis at twelve schools service students with educationaldifficulties run by the prefecture.
In the 2019 fiscal year, the drama and expression workshops held at
Tono High School developed into a course that the Faculties of
Pharmacy and Nursing of the Gifu University of Medical Science
now offers in partnership with Bungakuza. It has thus contributed to“relationship building” in community nursing and care of the elderly.
In these ways, I believe that working on evaluation is an investment
in order to share your activities with someone, and take another step
forward. Evaluation becomes an opportunity for communication to
evolve among those involved: clarifying the meaning and goals ofyour own activities provides hints about how to take your project
forward.

3 key points

In addition, SROI was originally a method created in order to see thesocial cost-effectiveness of large-scale projects, and the number ofexperts in evaluation for SROI is decreasing, so I do not think thatyou need to go out of your way to choose SROI as an evaluation
method.

We talked with Kisei Ei of ala!

﹈
SROI evaluation﹈
SROI evaluation

When we calculated the monetary value of the project outcomes,on the basis of a comparison of the pre- and post-workshop
questionnaires and the interviews, the results were as follows for an
average of two workshops each per year for around 620 children andyoung people:· Inputs = 2,100,000 yen· Outcomes = 4,851,777 yen (net present value)· SROI = 2.31
In addition, the results showed that the workshops play a particularly
useful role in self-expression, understanding others, andcommunity-building within classes, and that they also function as aplace of learning for teachers. For example, when we looked at thechanges in outcomes between the pre- and post-workshop
questionnaires, we found great improvements in the scores for items
such as “I communicate my thoughts and expressions”, “I havebecome more interested in relationships with other people”, and “Mycommunication skills have improved”.

be convincing to funders (government) if you have investigation
results saying, for example, that X % of people underwent a
particular change, showing the degree of contribution to SDGs, orthe importance of outcomes in terms of the government’s granddesign or overall plans.

﹇Four case studies﹇
Four case studies

Chika Ochiai

We talked with Chika Ochiai
of K-three Inc., who was involved
with this evaluation!

1. SROI is quite complex, so expert support is essential.
2. The social change brought about by the project is converted
into monetary value, which is used as evidence.
3. It is important to analyse why changes were or were notbrought about by the project, feeding the results back into a
logic model, and using them to improve the project.

61

Toride Art Project
and Outenin Temple

Arts Commission
Yokohama

Tokyo Bunka Kaikan

Kani City Cultural Creation
Center ala

Peer review evaluation

Empowerment-type project
improvement evaluation

Creating unique
indicators

SROI evaluation

Evaluation goals

Finding ways to carryout projects using
methods other than
third-party evaluation

Support implementing
organisations projects tofacilitate improvement, while
also understanding theimportance of evaluation of
their various activities.

Overview of
evaluation
activities

Implementing
organisations partnerwith organizations orcolleagues (peers) in thesame field to evaluateeach other’s projects

The grants secretariat gives
some workshops to encourage
self-evaluation by implementing
organisations. In addition,
selection panel members give
feedback before, during, andafter implementation.

What are the mainthings evaluated?

The project as a whole

External experts

Evaluation goals

Clarify the project’s value
and impact from an
objective standpoint,
leading to project
improvement andprogramme development

By calculating the monetary
value of social value, clarify
the meaning and outcomesof the project, and fulfil the
duty of accountability

Overview of
evaluation
activities

Researchers and theimplementing organisation
create indicators to affirm
the project’s uniquecharacter

Evaluation experts validate
the project and calculate
the monetary value of thechanges it has brought
about

TheoryWhat are the mainthings evaluated?

Outcomes

Outcomes

Unnecessary

Necessary

External experts

Necessary

Necessary

Who carries out
the evaluation?

Reciprocal evaluation
by others in the field(= peers)

Implementing organisations,experts in the arts andcommunity building

Who carries out
the evaluation?

Researchers in psychology
of aging and in thesociology of art

Experts in SROI
evaluation

Why it’s
recommended· You can discover yourproject’s value.
· It can be done without
experts.

· Organisation implementing the
project: You can discover yourproject’s value.
· Grants secretariat: You can fulfil
your duty of accountability
towards your parent
organisation, while also
encouraging the growth of theorganisations receiving funding.

Why it’s
recommended· You can identify the valueof your project visible tothird parties.
· You can carry out
evidence-based
programme development.

· You can make the value of
your project visible to third
parties.

Four case studies

Four case studies

62

List of case studies

63

Four case studies

Summary
The initiatives introduced in this chapter are just a few
of the many examples of evaluation in practice. Simply
by looking at these four cases, you can tell that each
evaluation method has its own distinctive characteristics and strengths.
Each of these initiatives is cutting-edge and unique, but
we hope that you have been able to see that in order to
get where they are now, the project members had tofeel uncomfortable with existing evaluation methods, orto try things which did not go well… Just like the cases
introduced in Chapter Two, they had to go through aprocess of repeated trial and error and to experimentwith various ingenious adjustments.
As we covered in Chapter One, when choosing anevaluation method, it is important to think hard about“when, with whom, and what” you want to think about
during the evaluation activities, and “what you want tocommunicate to whom.” Creating a cycle in which you
review the project after implementation and feed theseresults into the next project will surely provide clues tohelp you to find an evaluation method which is suitablefor you.

64

The first stepon the evaluation journey
In this chapter, we take a step back to think
about ingenious measures we can take tocreate a space for dialogue with various people
involved with a project. Through a space fordiverse dialogues, we will aim to gather ideas
to collectively articulate our project goals andtargets. How and with whom should we engage
in dialogue which leads to evaluation carriedout or the benefit of the community? Let’s lookat some tips together.

65

Right! I understand how to evaluate now,
so where should I start?
What were the goals of our project again…?

Project GDs who t h in g

ich are moaleani ng ful for s ocie ty !

That’s it!

actly does
What extice…?
n in practhat mea

Well, it’s… err… all kinds of things.
But I can’t really put them into words…

It’s difficult to put them into words,
isn’t it? Through dialogue witheveryone involved, you can gain new
insights and find new words!
Nice to meet you!

66

Throughout this book, we have discussed the importance of
thinking about your project’s goals and targets when designingevaluation and choosing evaluation methods. However, putting
the project’s goals and targets into words is no simple task. There
are probably many people implementing projects who find it
difficult to articulate project goals to colleagues, even if they havewritten them out in a funding application. It could be that project
goals have been set and activities carried out, but they do notseem to match the actual situation.
A repeated process of practice and dialogue is necessary to getcloser to finding the works to express the project’s goals andtargets. First, try to write down the goals and targets, even if thewords you use are temporary. Then, through dialogue, try to articulate the discoveries you have made through practice, or the times
when you have experienced or observed the project’s impacts.
Through this practice, you should be able to communicate about
the project in increasingly clearer terms.

The first step on the evaluation journey

The first step on the evaluation journey

Creating a space for dialogue

With whom, then, should dialogue take place, and in what way?
In this chapter, we will look in concrete terms at dialogue partners
and points for making dialogue go well. Creating a space fordiverse forms of dialogue with the people involved is the key tofinding words to describe your project.

Hello!

67

What kinds of dialogue partners are there? As well as event participants, we can think of various people involved with the project:funders, operational staff, artists, partner organisations, and soon. The diagram below is an example showing how the organisation in charge of a particular project communicated with partners
in the form of a “Stakeholder Map”.
Think about with whom and in what way you could talk in order to
get the information and perspectives you need for your own
project, and what kinds of people are involved, including names,
if possible.

Encouraging dialogue demands ingenuity. There are many
creative ways to obtain the information and words that you need,including the use of questionnaires, photos, or videos, or reading
reports on similar projects. Also, dialogue using words or letters
can sometimes be difficult in socially inclusive arts activities. In
such cases, it is important to stand in your partner’s shoes for a
moment and co-create a form of dialogue which will be easy forthem to participate in.
So what kinds of creative means can you use to create a space for
dialogue? In the next pages, we will look at some concrete
methods.

The first step on the evaluation journey

The first step on the evaluation journey

Who do we need to talk to? How should we go about it?

Stakeholder Map(staff of divisions concerned with cultural,
welfare, or educational projects in central
government ministries, prefectural
governments, or municipal governments)

Government funders

Intermediary support organisations(arts councils, foundations promoting the arts and culture, etc.)

Funders
Experts

Families of the participants(experts in evaluation, clinical psychologists,
experts in fields related to the project such as welfare,the arts, education, etc.)

People/organisations
implementing the project

Project’ s core
participants(Project organisers,
whether individuals or groups)

Coordinators/ facilitators
Artists
Supporters of the participants(NPO staff, care staff, schools, etc.)

Collaborating organisations(sign language clubs, information accessibility apps, venue owners, etc.)

Partner organisations
Operational staff

68

Support staff(paid/unpaid volunteers, local community members, etc.)

Event participants

69Organise objective information

First, create a framework to open up dialogue. Most projects are run
by a limited number of people who have their hands full simply
carrying out the project itself. However, even if you obtain information and verbal feedback through dialogue, it is hard to use it in yourproject if you cannot share these with colleagues. By exercising
ingenuity when carrying out your existing work, create an environment in which you can hold regular internal discussions about
project goals and impacts.
For example, you could make use of your organisation’s regularmeetings. It is hard to arrange a separate meeting to discuss yourproject’s goals, but try to set aside even a short time initially to
discuss them during the regular meetings. Probably there will bedays when you cannot spare even 10 minutes, and days when you
can talk about them for 30 minutes or more, but it is important tohave an ongoing discussion.
Creating an opportunity for people from different backgrounds whoare involved with the project to get together and hold a discussion is
also effective. Choose a time when this is easy to implement, suchas at the start or end of the fiscal year, or mid-term, and put it into
your annual schedule.
In the next section, we will look at creative ways to obtain essential
information and words through dialogue under the following
headings.
O rg an is e ob

je ct iv e in fo

rm at io n

P. 71

Ad m in is te r

List en to peo ple dire ctly (inteR e c o rd70

w h a t ta

k e s p la

ce

P.7 6

qu es tio nn ai

rvie ws)

re s

P.74

Carry out a review

P.77

P. 72

Try to gain a good idea of the objective facts about the projects
which you are carrying out (or have carried out). Gather information
about, for example, participant numbers, demographics, previous
participation histories (number of years participating), attributes, andso on. You can consider project inputs (investments) as well as
outputs (results).
Even if something is not apparent from one year’s data, you may beable to observe changes over time if you continue to collect data. By
putting into words the things which you notice when looking at thedata, you can obtain clues helping you to grasp the impact of the
project.

I’ ve looked at this year’ s data,
but there don’ t seem to be any
particular changes…

The first step on the evaluation journey

The first step on the evaluation journey

Facilitating dialogue

When I put together the data overthree years, I understood how
various things are changing…

Three
years

Try looking over the data together with the people involved andtalking about what you can see from it.
What are the figures which particularly intrigue you? Whatchanges can be seen over time?
What kinds of significance do you think these figures orchanges have?

71Use questionnaires to collect participants’ feelings about events.
It is a good idea to talk with the people involved and decide whatkinds of thing you want to know when planning what the questionnaire should cover. By doing so, you can choose whether written
answers (qualitative data) or check boxes or other evaluation
scales (quantitative data) are the most appropriate method foreach item. It is also important to use your ingenuity to makeanswering the questionnaire easy for those whose opinions you
want to hear.

I made a participant questionnaire,
but afterwards, various people told
me they’ d like to include this or that,and so it’ s hard to come up with afinished questionnaire…

Think about the questions in two stages
In order to find out what the people involved with running the
project would like to hear from the participants, you could
first carry out a questionnaire internally. Based on this, you
could draft a questionnaire for distribution to the participants,
then discuss this at an executive committee meeting or
similar, and include everyone's input in the final questionnaire.

This method is effective when there are alarge number or a broad range of people
involved. Although it may seem like morework, it saves effort in the end!

72

Expressions which are suited to the respondents
When administering a questionnaire to young children orothers who struggle to read, it is a good idea to take care touse simple expressions and words which are easy to read.
Questions using five-point scales such as “Very good/ Good/
Average/ Bad/ Very bad” are abstract, and so they can be
hard for respondents such as children to understand. If you
ask about their feelings using smiley faces or other illustrations, it may be easier for them to answer.
Circle the way you feel now

The first step on the evaluation journey

The first step on the evaluation journey

Administer questionnaires

Administering questionnaires before and after an event
In ongoing activities, you may be able to understand theeffectiveness of an event through the changes in the attitudesof the participants before and after taking part. One way is toadminister questionnaires before and after holding the event.
Another is to send a follow-up questionnaire, for example byemail, a certain time after the event.
Questionnaire before an event

Questionnaire after an event

1. What made you want to participate
in this event?
2. What kind of image (impression) do
you have about the topic of thisevent?

1. Did your image (impression) about
the topic of this event change?
2. Has your thinking about how you
would like to engage with the topic
changed?

Administering a questionnaire is not the end. Share what you
learned from the information gained among everyone involved,
and make use of it in order to improve the project.
What intrigues you most among the responses to thequestionnaire?
Tell your colleagues why this intrigues you, and listen to what
intrigues them.

73Listen directly to how event participants and those involved felt inorder to gather their responses. To obtain opinions about the
project from many angles, it is a good idea to listen to people inseveral different positions. You can interview them individually orin groups. Talk with others involved beforehand to decide what
you would like to ask.

Think about the order in which you would like to cover topics
Start with the stage before participation in the event and proceedin chronological order, asking about what prompted participants’thoughts or feelings to change, and in what ways.
1. Thoughts and feelings before participating: “What did you
expect to get out of participating?” “What did you think
about XYZ before taking part?”, etc.
2. Realisations while participating: “What incident or remark
made the greatest impression on you during the event?
Who was involved, and what did they say or do? Please tell
us why it made an impression on you”, etc.
3. Changes in ways of thinking or perceptions after participation:“How did your understanding of XYZ change?” “Did you
learn or realise anything new which will influence your daily
life or activities from now on? If so, what?”
The testimonies obtained in this way will of course serve as
useful information when designing evaluation, but are alsovaluable evidence for evaluation itself. This can also be
applied to the creation of questionnaires.

74

Share opinions written on sticky notes
You can also capture what participants thought during an event,or what they wanted to learn more about, by having them writethese on sticky notes.
By having participants write as
many thoughts as possible,
emphasising quantity overquality, and setting up times
for group discussion based onthese notes, everyone canshare their thoughts and join in
the discussion, whether they
are a loud or a quiet person.

Opinion Card

Please write down yourthoughts on today's lecture
on a sticky note!

- What did it make you think about?
- What do you want to learn more about?
I thought about...

I want to learn
more about...

The first step on the evaluation journey

The first step on the evaluation journey

Listen to people directly

Listen to the voices of those for whom responding toquestionnaires is hard
Even if you ask in a questionnaire “What was the most fun?”,it can sometimes be difficult to remember anything. It might
be a good idea to make some creative adjustments so that it
is easier for people to answer, such as showing photographs
which clearly capture the nature of the event as you ask themto choose the most fun part, or ask for their opinions.
By talking directly with people, you can ask penetrating
questions about the effectiveness of the event or about issues in
the course of conversation. In addition to words, the respondents’ facial expressions and body language also provide clues
about their feelings.
Which response by which person made the greatest impression on you during the interview?
Talk with your companions about the significance of this
response to dig deeper.

75

Carry out a review

Record what takes place during project management and when
you hold events. For example, using photos or videos to record
participants’ expressions and the scenes at the venue is effective.
Using these records, you can also put the feelings of those
involved and participants into words and share them. Please see
the reference materials for detailed archival methods.
Referencematerials

p. 81, ⑳

Keep a record through videos
At workshops and other events involving physical movement,take videos showing the activities. It is fine to do this fromone fixed standpoint. This is useful in showing what triggered
unexpected events during the activities, and understanding
the context in which they occurred. Watching the videos or
looking at photos together and writing down and sharing
anything you notice can also lead to fresh discoveries.
What I noticed
The situation which intrigued you

Why did it intrigue you?

Through records, you can reconfirm the facts, and share what
happened at the event with everyone involved.
Did anything unplanned or unexpected happen?
If so, what? What was the significance of this occurrence?
Discuss with your colleagues.

76

Review what happened during project management and whenholding events with the people involved with your project. Becoming aware of what you felt or discovered during the activities andsharing this with colleagues will likely lead to new insights.
Creating a space which generates positive thinking
A review and a reflection are two different things. Make aconscious effort to avoid listing all the shortcomings as material for reflection, but rather to conduct a review which unlocks
motivation. For example:
・Have each person comment on one thing that went well.
・Build on each other’s observations to talk further: “That’s
right; and this happened, too…”

Referencematerials

The first step on the evaluation journey

The first step on the evaluation journey

Record what happens

p. 81, ㉒

The fuzzy curve
At art events, people often find themselves in a “fuzzy state”,
where something is on their mind but they cannot put it into words.
It could be a good idea to try representing these changing feelings
as a line graph (feelings curve). One
Feeling fuzzy
method is to ask people to write on the
When we did XYZgraph what caused this “fuzzy” feelingto peak during the event, and to share
Feeling clear
their answers with each other.
Try to find creative ways to avoid formal or serious reflections,but instead carry out an enjoyable review. Tea or sweets,
images or videos help to liven things up.
What intrigued you most among other people’s opinions?
Using this as a starting point, tell each other about yournew insights.

77

The first step on the evaluation journey

Summary
In order to practice a way of evaluation which is “for you”,
which “energises you”, rather than a one-way type basedon the indicators imposed by your funder, it is essential tobe able to express your project’s goals in your own words.
This is no easy task, but by putting what you have learned
through practice into words via opportunities for the diverse
types of dialogue which we have seen in this chapter, you
can gather a wide range of data.
Becoming able to express the project’s goals and significance in your own words will lead people from a variety ofbackgrounds to grasp the project’s vision and get involved.
In addition, while each person plays their role in their particular situation, they will build relationships with each other,creating the foundations for the project to progress in the
direction envisaged.
When we asked the trailblazers in the evaluation of culturalprojects about the effects of their evaluations, they replied:“Our relationships with everyone involved have changed,
making communication easier.” Evaluation also functions as
a hub for the promotion of mutual communication between
the various people in different positions who are involved
with a project. The spaces for dialogue created with those
involved during the evaluation process also serve as places
to build relationships with colleagues who will improve anddevelop the project together, taking it forward into the future.

78

Members of the expert committee
These are the people who contributed to the production of this handbook by
participating in the interviews for the case studies in Chapter 3, held between
September 2020 and February 2021, as well as in two meetings of an expert
committee and as content supervisors. Roles and affiliations listed are those
held at the time of participation.
Kisei Ei

Director and General Theatre Manager, Kani Public Arts Center ala

Chika Ochiai

Director and Chief Knowledge Officer, K-three Inc.; Researcher, Graduate
School of Media and Governance, Keio University

Sumiko Kumakura

Professor, Graduate School of Global Arts, Tokyo University of the Arts

Eisuke Sugizaki

PR & ACY group, Yokohama Arts Foundation

Yukiyo Sugiyama

Chief of Inclusion & Partnership Section, Planning and Production Division,
Tokyo Bunka Kaikan

Asako Hirokawa

Chair, Theatre Accessibility Network (TA-net), NPO

Interviewees
These are the people who cooperated with the production of this handbook by
participating in the interviews for the case studies in Chapter 3, held between
September 2020 and February 2021. Roles and affiliations listed are those heldat the time of interview.
Yasue Habara

Board Member and Secretary General, Toride Art Project

Ayako Omuka

Deputy, Toride Art Project

Saya Makihara

Doctoral candidate in Sumiko Kumakura’s research group, researching
Artistic Creativity and the Environment within Musicology and Music Studies,
Graduate School of Music, Tokyo University of the Arts

79

Reference materials
These publications and websites formed the basis of the contents of each chapter.
(URLs are current as of February 2021).
■Publications in Japanese① Sumiko Kumakura (supervisor and editor); Saya Makihara, Yuriko Minamoto,

Tomoko Wakabayashi (special contributors), “Peer review of arts projects: an evaluation method using dialogue and mutual support” (Suiyosha, 2020)② Yuriko Miyamoto and Iwao Oshima (editors), Kiyoshi Yamaya (supervisor),“Programme evaluation handbook: Fundamentals and application of evaluation
methods aimed at solutions to social issues” (Koyo Shobo, 2020)③ Vijay Kumar (author), Noriko Watanabe (translator), “101 design methods: A

structured approach for driving innovation in your organization” (Eiji Press, 2015)■Project reports, websites④ & Geidai

http://ga.geidai.ac.jp/and-geidai⑤ Toride Art Project

https://toride-ap.gr.jp⑥ Arts Commission Yokohama
https://acy.yafjp.org⑦ YokohamArtLife

https://yokohamartlife.yafjp.org⑧ Yokohama Arts Foundation “Report on YokohamArtLife Leading Programme,

Special Support Project for the Arts and Creativity”
https://yokohamartlife.yafjp.org⑨ Tokyo Bunka Kaikan “Workshop! 2020 on stage & legacy, aiming for a society inwhich all people can interact through music”
https://www.t-bunka.jp/about/on_stage.html⑪ Tokyo Bunka Kaikan “Validation of music workshops for senior citizens (FY 2018)”
https://www.t-bunka.jp/stage/2973⑫ Tokyo Bunka Kaikan “Validation of music workshops for senior citizens (FY 2019)”
https://www.t-bunka.jp/stage/5980⑬ Tokyo Bunka Kaikan, introductory video of music workshops for senior citizens
https://youtu.be/DkLqjIhqRks⑭ Tokyo Bunka Kaikan “Report on research investigating socially inclusive initiatives using art”
https://www.t-bunka.jp/about/pdf/research.pdf⑮ Tokyo Bunka Kaikan “Annual report”

https://www.t-bunka.jp/pdf/annual_report.pdf⑯ Tokyo Bunka Kaikan Channel

https://www.t-bunka.jp/info/5492/⑰ Kani City Cultural Creation Center ala
https://www.kpac.or.jp/ala/⑱ Kani City Cultural Creation Center ala “Report on an investigation to validate the efficacy ofutilising the creativity of culture and the arts”

https://www.kpac.or.jp/data/report/sroi_houkoku2018.pdf⑲ Investigation commissioned by the Cabinet Office “Final report on research investigating theevaluation of social impact”

https://www.npo-homepage.go.jp/uploads/social-impact-hyouka-chousa-all.pdf⑳ Arts Council Tokyo (Tokyo Metropolitan Foundation for History and Culture) “Arts Archive Guide”
https://tarl.jp/library/output/2015/art_archive_benricho/㉑ Arts NPO Link “Practical guide to evaluation born on the frontlines of arts practice”
http://arts-npo.org/img/artsnpodatabank/ANDB2018-19.pdf㉒ Fukuoka City “Guide to working with Fukuoka City’s community centres”

https://www.city.fukuoka.lg.jp/shimin/kominkanshien/business/fukuokasikouminkantunaginotechou.html⑩ Tokyo Bunka Kaikan “Guidebook to the Art Activities leading to Social Inclusion”
https://www.t-bunka.jp/about/pdf/tbk_guidebook.pdf

8081

Research team members

Conclusion

Members of the Agency for Cultural Affairs and Kyushu University Joint

Evaluation is something done not for others, but for oneself.
This is something which has been brought home to me throughout the creation of thishandbook. When we talk about evaluation, we tend to emphasise the part in which“the teacher checks your work afterwards”, like marking a test, but the important partis “making use of the results of this check to improve your way of studying”.
When we researched what was actually happening, we found that where evaluationand the projects themselves are carried out in an integrated way, both evaluation andthe projects go well. However, where the two are detached from one another, neither
goes well. If you can carry out evaluation for your own sake, the project contents will
get better and, as a result, society will also come to value it more highly. It seems
obvious when we think about it, but somehow, this point tends to be forgotten.
There are two reasons for this. One is the existence of preconceptions about evaluation. We have looked at these in detail in this handbook. The other reason, which was
not dealt with much in this handbook, is that there are problems with project designon the part of government and other funders.
The project design of grants, financial support, and outsourcing, too, needs to beplanned with attention to detail, based on a deep understanding of the evaluation ofcultural projects and of the potential of socially inclusive arts activities, as well as aconsideration of the actual needs and circumstances. In addition, evaluation needs tobe put to use to improve project design. However, in many cases, both designand evaluation flounder because of the issues we have highlighted here. This isa challenge for the future.

The Agency for Cultural Affairs
Yuki Asakura

Researcher, General & Policy Research Group, Headquarters for Vitalizing Regional
Cultures, the Agency for Cultural Affairs

Tokuji Saidera

Senior Researcher, General & Policy Research Group, Headquarters for Vitalizing
Regional Cultures, the Agency for Cultural Affairs

Natsuko Shirako

Chief, General & Policy Research Group, Headquarters for Vitalizing Regional Cultures,
the Agency for Cultural Affairs

Kyushu University Social Art Lab
Mia Nakamura

Associate Professor, Kyushu University Faculty of Design(Lead researcher, Author of Chapters 1 & 3)

Yuichiro Nagatsu

Assistant Professor, Kyushu University Faculty of Design(Author of Chapter 2)

Tsukasa Muraya

Research Fellow, Kyushu University Faculty of Design(Author of Chapter 4)

Akemi Minamida

Post-Doctoral Fellow, Japan Society for the Promotion of Science,
based at Kyushu University Faculty of Design (Author of Chapter 3)

Advisers

Torao Osawa

Lead Researcher, Arts and Culture Projects Office, NLI Research InstituteThis publication brings the “Social Inclusion through the Arts and Culture” handbook
series to a close. We received extensive support and collaboration from many people
throughout the process, and I would like to express my deep gratitude to all of them.

Satoshi Miyata

Office Manager, donner le mot NPO (Editor)

Mia Nakamura

Kana Sakurai

Staff, donner le mot NPO (Author and editor of Chapter 3)

Megumi Watanabe

Staff, donner le mot NPO (Worksheet designer and editor)

Lead Researcher, the Agency for Cultural Affairs of Japan & Kyushu University Joint Research Team
Associate Professor, Faculty of Design at Kyushu University

82

Research Team

Secretariat

83

Let’s do it!

Social Inclusion Through Culture and the Arts:
A Practical Evaluation Handbook

The voices of many come together to create new values in our society

Date of Japanese publication: 31.3.2022
Created by the Agency for Cultural Aﬀairs of Japan & Kyushu University Joint Research Team
(Lead Researcher: Mia Nakamura)

Authors: Mia Nakamura, Yuichiro Nagatsu, Tsukasa Muraya, Akemi Minamida, donner le mot (NPO)English translation:
English editing:
Editing:
Design:Emma Parker

Yaya Yao, Mia Nakamura

donner le mot NPO
Kaori Nagasue

Published by Social Art Lab, aﬃliated with the Faculty of Design, Kyushu University
4-9-1 Shiobaru, Minami Ward, Fukuoka City, 815-8540
http://www.sal.design.kyushu-u.ac.jp• This handbook is based on the results of a joint research project between the Agency for Cultural Affairs and universities/research institutes entitled “Social Inclusion Through the Arts and Culture: Creation of a Project Evaluation Handbook.”• This handbook can be downloaded in PDF at http://www.sal.design.kyushu-u.ac.jp/publications.html.
• Permission is given for the use (reprinting, copying, sharing, etc.) of this handbook for non-commercial purposes only,
and on condition that the source is stated.
© 2022 The Agency for Cultural Affairs & Kyushu University Joint Research Team