COMMISSION STAFF WORKING DOCUMENT Ex-post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta) Accompanying the document Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions Ex Post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta)
Date: 2020-08-18
Institutions: Directorate-General for Education, Youth, Sport and Culture, European Commission
Subjects: EU action, European cultural event, European symbol, Friesland, Malta, Netherlands, capital city

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 18.8.2020  SWD(2020) 165 final  COMMISSION STAFF WORKING DOCUMENT Ex-post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta) Accompanying the document Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions   Ex Post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta)    {COM(2020) 386 final}  
 
 
Table of Contents 

1. 

2. 

3. 

INTRODUCTION ................................................................................................................................. 3 

BACKGROUND TO THE INTERVENTION ..................................................................................... 4 

IMPLEMENTATION / STATE OF PLAY .......................................................................................... 7 

4.  METHOD ............................................................................................................................................ 11 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ........................................... 13 

6. 

CONCLUSIONS ................................................................................................................................. 23 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 25 

1. 

LEAD DG, DECIDE PLANNING/CWP REFERENCES ............................................................. 25 

2.  ORGANISATION AND TIMING ................................................................................................... 25 

3. 

4. 

EXCEPTION TO THE BETTER REGULATION GUIDELINES .............................................. 25 

EVIDENCE, SOURCES AND QUALITY ...................................................................................... 25 

ANNEX 2: STAKEHOLDER CONSULTATION ....................................................................................... 27 

ANNEX 3: METHODS AND ANALYTICAL MODELS ........................................................................... 28 

1 

 
 
 
 
 
 
COMMISSION STAFF WORKING DOCUMENT 

Accompanying the document 

Report from the Commission to the European Parliament, the Council, the European 
Economic and Social Committee and the Committee of the Regions  

Ex Post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta) 

1. 

INTRODUCTION 

This  document  describes  the  methodology  and  findings  of  the  ex-post  evaluation  of  the 
European Capital of Culture (ECOC) Action for 2018.  
Article  12  of  Decision  No  1622/2006/EC1  (here  after,  the  “Decision”)  requires  that  the 
Commission ensures the external and independent evaluation of the results of the ECOC from 
the  previous  year.  The  aim  of  the  evaluation  is  to  better  understand  how  the  two  European 
Capitals of Culture of the previous year contributed to the objectives of the Action, whether 
they  achieved  their  objectives  and  whether  the  implementation  proceeded  in  line  with  their 
original  application.  It  is  also  to  contribute  to  reinforcing  the  existing  evidence-base  on  the 
ability of the Action to produce cultural, social and economic impact. Finally, the objective is 
to draw lessons, conclusions and recommendations that may be useful for Union institutions, 
future ECOC or cities wishing to bid for the ECOC title in the future. 

The  Commission  started  these  yearly  evaluations  with  the  2007  title-holders  and  the 
evaluation  of the ECOC 2018 is  the latest  to  date in  a series of 12 similar  yearly  exercises. 
This staff-working document is the fourth covering the ECOC Action in as many years, while 
the  Action  in  itself  has  not  changed  in  the  period  considered,  with  only  the  pair  of  cities 
hosting the Action being different each year. 

This  staff-working  document  summarizes  the  findings  of  the  external  evaluation  of  the 
implementation of the ECOC 2018, including the selection and monitoring procedures as well 
as  the  operational  delivery  by  the  two  hosting  cities,  namely  Valletta  in  Malta  and 
Leeuwarden in the Netherlands2. The evaluation examined how the two cities developed their 
respective  applications  and  cultural  programmes,  how  they  delivered  the  planned  initiatives 
throughout the year and any cultural, general and longer-term impact generated by the Action. 

The  evaluation  constitutes  a  valuable  opportunity  to  reconsider  critically  the  past  year  with 
the  intention  of  collecting  further  insight  and  lessons  based  on  the  experiences  of  the  host 
cities. 

1   Decision  No  1622/2006/EC  of  the  European  Parliament  and  of  the  Council  of  24  October  2006  establishing  a 
Community Action for the European Capital of Culture event for the years 2007 to 2019 (OJ L 304, 3.11.2006, p. 1), 
available at:  

- http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2006:304:0001:0006:EN:PDF. 

2  

document 

Full 
01aa75ed71a1/language-en. 

available 

at: 

https://op.europa.eu/en/publication-detail/-/publication/6312a17a-1b6a-11ea-8c1f-

2 

 
 
 
                                                           
It  is  also  important  to  underline  that  the  aim  of  the  yearly  evaluation  is  not  to  lead  to  any 
change in  the regulation governing the ECOC Action.  Indeed, those rules changed in  20143 
following  an  ad  hoc  interim  evaluation  of  the  selection  and  monitoring  procedures  of 
European Capitals of Culture4, but the new rules only apply for cities designated as ECOC for 
the  years  from  2020  to  20335.  It  would  therefore  be  premature  to  use  the  outcomes  of  the 
evaluation of the two ECOC 2018, governed by previous Decision No 1622/2006/EC, to draw 
conclusions  related  to  Decision  No  445/2014/EU,  which  introduced  changes  in  particular 
regarding  the  selection  procedure,  monitoring  arrangements  or  the  payment  of  the  Melina 
Mercouri Prize6. 

2.  BACKGROUND TO THE INTERVENTION 

2.1.  The European Capital of Culture Action 

The initial scheme of "the European City of Culture" started at an intergovernmental level in 
19857  upon  an  idea  of  the  then  Greek  Minister  of  Culture,  Melina  Mercouri.  The  scheme 
recognized  Europe  as  a  centre  for  artistic  development,  underpinned  by  an  exceptional 
cultural richness and diversity, with cities playing a vital role in the formation and spread of 
cultural expressions. 
In  1999,  Decision  1419/1999/EC  of  the  European  Parliament  and  the  Council8  gave  the 
scheme the status of a Community Action under the name of "European Capital of Culture" 
and set up a more predictable, consistent and transparent system for the designation of hosting 
cities. Its legal foundation was Article 151 of the Treaty (now Article 167), which calls on the 
Union to “contribute to the flowering of the cultures of the Member States, while respecting 
their  national  and  regional  diversity  and  at  the  same  time  bringing  the  common  cultural 
heritage  to  the  fore”.  The  Decision  introduced  new  selection  procedures  and  evaluation 
criteria while Member States were ranked in a chronological order of entitlement to host the 
event each year.  

3   Decision No 445/2014/EU of the European Parliament and of the Council of 16 April 2014 establishing a Union action 
for the European Capitals of Culture for the years 2020 to 2033 and repealing Decision No 1622/2006/EC (OJ L 132, 
3.5. 2014, p. 1), available at: 

- http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2014.132.01.0001.01.ENG. 

4   Available  at:  https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/library/2011-capitals-culture-

assignment-report_en.pdf. 

5   Article 17 of Decision No 445/2014/EU. 

6   The new Decision, which covers the ECOC titles 2020 to 2033, retains the general structure and main elements of the 
previous Decision but introduced improvements to maximise the benefits of holding the title as well as taking part in the 
competitive  process  for  all  bidding  cities  and  their  citizens.  Improvements  include  among  others  the  introduction  of 
more explicit and measureable criteria, the reinforcement of conditionality for the payment of the Melina Mercouri Prize 
and the obligation for the cities – instead of the Commission – to carry out the ex-post evaluation of the ECOC year. 

7   Resolution of the Ministers responsible for Cultural Affairs, meeting within the Council, of 13 June 1985 concerning the 
annual event 'European City of Culture' (85/C 153/02), on the initiative of the former Greek Culture Minister, Melina 
Mercouri.  

8  Decision  1419/1999/EC  of  the  European  Parliament  and  of  the  Council  of  25  May  1999  establishing  a  Community 

action for the European Capital of Culture event for the years 2005 to 2019 (OJ L 166, 1.7.1999). 

3 

 
 
                                                           
 
In  2005,  the  European  Parliament  and  the  Council  adopted  Decision  649/2005/EC9  to 
integrate the ten countries that joined the European Union in 2004. 

In 2006,  a new  Decision  – Decision 1622/2006/EC  – entered into force.  This  Decision kept 
the principle of a chronological order of Member States but further refined the objectives of 
the  Action 
arrangements. 
Decision 1622/2006/EC is the legal basis governing the European Capital of Culture event for 
the titles 2007 to 2019. 

and  monitoring 

introduced 

selection 

new 

and 

As  mentioned  under  section  1,  in  April  2014,  the  European  Parliament  and  the  Council 
adopted  a  new  Decision,  but  cities  designated  as  ECOC  for  the  years  up  to  2019  continue 
being regulated by Decision No 1622/2006/EC.  

The  2014  Decision  was,  in  turn,  amended  in  September  2017  to  open  the  ECOC  Action  to 
European Free Trade Association countries that are parties to the Agreement on the European 
Economic Area10. 

2.2.  Objectives of the ECOC Action 

The ECOC Action aims to highlight the richness and diversity of European cultures and the 
features  they  share,  thereby  promoting  greater  mutual  understanding  among  European 
citizens, as well as to foster the  contribution of culture to the long-term development of the 
cities. ECOC shall strive to foster cooperation between cultural operators, artists and cities in 
Europe,  foster  the  participation  in  cultural  activities  of  the  citizens  living  in  the  city  and 
surroundings while raising the interest of citizens from abroad, to be sustainable and to be an 
integral part of the long-term cultural and social development of the city. 

The hierarchy of objectives presented in the table below is based on the objectives as stated in 
Decision  No 1622/2006/EC,  but  it  has  been  updated  to  reflect  the  content  of  the  new  legal 
basis  for  ECOC  post-2019.  The  general  and  strategic  objectives  are  taken  directly  from 
Article 2 of Decision No 445/2014/EU, with the operational objectives flowing logically from 
these.  They  are  also  informed  by  the  selection  criteria  detailed  in  Article  5  of  the  2014 
Decision.

9   Decision  649/2005/EC  of  the  European  Parliament  and  of  the  Council  of  13  April  2005  amending  Decision 
No 1419/1999/EC establishing a Community action for the European Capital of Culture event for the years 2005 to 2019 
(OJ L 117, 4.5.2005). 

10   Decision (EU) 2017/1545 of the European Parliament and of the Council of 13 September 2017 amending Decision No 
445/2014/EU establishing a Union action for the European Capitals of Culture for the years 2020 to 2033 (OJ L 237, 
15.9.2017, p. 1). Text available at: 
http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017D1545. 

4 

 
 
                                                           
Table on ECOC hierarchy of objectives 

General objective 

Safeguard and promote the diversity of cultures in Europe, highlight the common features they share, and foster the contribution of culture to the long-term development 
of cities 

Specific objectives (SO) 

SO1:  Enhance  the  range,  diversity  and  European 
dimension of the cultural offer in cities, including 
through transnational co-operation 

SO2:  Widen  access  to  and  participation  in 
culture 

SO3:  Strengthen  the  capacity  of  the 
cultural  and  creative  sector  and  its 
links with other sectors 

SO4: Raise the international 
profile  of  cities 
through 
culture 

Operational objectives 

Stimulate  a  diverse  range  of  cultural  activities  of 
high artistic quality  

Create new and sustainable opportunities for 
a  wide  range  of  citizens  to  attend  or 
participate in cultural events 

Implement  cultural  activities  promoting  cultural 
diversity, dialogue and mutual understanding 

Implement cultural activities highlighting (shared) 
European cultures and themes  

Involve  local  citizens,  artists  and  cultural 
organizations 
and 
implementation 

development 

in 

Improve cultural infrastructure 

Develop 
governance of the cultural sector 

skills,  capacity  or 

the 

Stimulate 
operation with other sectors 

partnership 

and 

co-

Attract  the  interest  of  a 
broad 
and 
European 
international public  

Involve  European  artists,  promote  cooperation 
transnational 
with  different 
partnerships  

countries 

and 

Provide  opportunities  for  volunteering  and 
foster links with schools and other education 
providers 

Combine  traditional  art  forms  with 
new types of cultural expression 

5 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3. 

IMPLEMENTATION / STATE OF PLAY 

3.1.  The selection and monitoring of the European Capitals of Culture 2018 

In accordance with Decision No 1622/2006/EC, Malta and the Netherlands were the two 
Member States entitled to host the ECOC in 2018.  

Under  this  Decision,  host  countries  are  responsible  for  the  procedure  leading  to  the 
selection of one of their cities as "European Capital of Culture". This is done through an 
open  competition  within  the  Member  State  concerned.  Against  this  background,  the 
relevant  managing  authorities  of  Malta  and  the  Netherlands,  respectively  the  Maltese 
Ministry of Tourism, Culture and the Environment and the Dutch Ministry of Education, 
Culture and Science, run parallel competitions, which both started with the publication of 
a call to trigger applications from interested cities.  

The selection is then in two phases: a pre-selection round (candidate cities are reduced to 
a  short-list)  followed  by  a  selection  round  (one  city  is  recommended  for  the  title).  A 
panel of thirteen members – six of whom nominated by the Member State concerned and 
the  other  seven  by  European  Union  institutions  and  bodies  (European  Parliament, 
Council,  Commission  and  Committee  of  the  Regions)  –  examines  the  bids  from 
candidate cities based on the objectives and criteria laid down in the Decision. 

Malta decided to start its selection procedure one year ahead of the normal time schedule 
because it knew that major infrastructure projects – which need long planning periods – 
were necessary to help  the country host  an event of the scale  and scope  of a European 
Capital  of  Culture.  It  therefore  published  its  call  for  submission  of  applications  in 
December 2010, seven years before the start of the ECOC-year while the rule is to start 
the competition “no later than six years before the event is due to begin”.  

The  main  stakeholders  at  national  and  local  levels  had  taken  an  early  decision  that  an 
application would be submitted on behalf of Valletta but involving the entire territory of 
the  Maltese  islands.  Valletta’s  application  was  consequently  the  only  one  submitted  by 
the call’s deadline of 17 October  2011.  Following the pre-selection meeting in  January 
2012,  the  city  was  short-listed  and  in  November  2012,  the  panel  recommended  that 
Valletta be awarded the ECOC 2018 title in Malta11. 

The Netherlands published their call in December 2011, in line with the normal time line. 
Five cities entered the competition by the call’s deadline of 31 October 2012: Eindhoven, 
Leeuwarden,  Maastricht,  The  Hague  and  Utrecht.  At  the  pre-selection  meeting  in 
November  2012,  three  of  them  (Eindhoven,  Leeuwarden  and  Maastricht)  were  short-
listed.  In  September  2013,  the  panel  recommended  that  the  ECOC  2018  title  in  the 
Netherlands be awarded to Leeuwarden. 

The  Council  of  the  European  Union  formally  designated  Valletta  and  Leeuwarden  as 
ECOC 2018, respectively in May 201312 and in May 201413. 

11  All 

panel’s 

pre-selection, 

selection 

and 

monitoring 

reports 

are 

available 

at: 

http://ec.europa.eu/programmes/creative-europe/actions/capitals-culture_en.htm. 

12  Council Decision No 2013/286/EU of 17 May 2013 designating the European Capital of Culture for the year 2017 
in Denmark and in Cyprus and the European Capital of Culture for the year 2018 in Malta (OJ L 162, 14.6.2013). 

6 

 
 
 
                                                           
In  line  with  Decision  No  1622/2006/EC,  once  designated  as  ECOC  and  until  the  title-
year,  the  two  cities  had  to  adhere  to  a  monitoring  procedure  directly  managed  by  the 
Commission.  

All along this phase, the progress in the cities' preparations was monitored and guided by 
a  panel  composed  of  the  seven  independent  experts  appointed  by  the  European  Union 
institutions  and  bodies,  which  also  checks  compliance  with  the  programme  and 
commitments  on  the  basis  of  which  the  two  cities  had  been  selected.  As  part  of  this 
process,  Valletta  and  Leeuwarden  submitted  progress  reports  and  attended  two  formal 
monitoring  meetings  with  the  panel  that  the  Commission  convened  in  November  2015 
and  April  2017.  On  top  of  that,  the  Commission  organized  informal  post-designation 
meetings  (that  do  not  result  in  public  reports,  as  they  do  not  have  a  formal  status) 
between the panel and Valletta in October 2013 and in November 2014 and between the 
panel and Leeuwarden in November 2014. An in situ visit (whereby a delegation of the 
panel and staff members of the Commission visit a hosting city to get a better insight of 
the development of the project) took place in Valletta in January 2016. No similar visit 
was  deemed  necessary  in  the  case  of  Leeuwarden  as  preparation  progressed  more 
smoothly.       

During this monitoring phase, the panel made a number of recommendations to the two 
cities, in particular to ensure that they adequately involve all relevant groups of citizens, 
develop an ambitious European dimension and reflect on a proper legacy of their ECOC-
year.  

As  was  the  case  for  all  previous  ECOC,  during  the  development  period,  the  two  cities 
introduced  modifications  into  the  programme  described  in  their  original  applications, 
either  in  response  to  a  changing  environment  or  to  the  recommendations  made  by  the 
panel.  

However,  the  changes  made  by  Valletta  were  of  a  much  bigger  scale  than  the  usual 
practise, as the initial concept of the bid was dropped following a change of leadership 
after  the  country’s  2013  general  election.  The  panel  took  note  of  these  changes  and 
underlined the importance to preserve the artistic coherence of the project in line with the 
application’s original objectives. At the last meeting, while praising Valletta for the good 
preparatory  work  in  terms  of  monitoring,  evaluation  and  involvement  of  the  various 
communities in the Maltese islands, the panel highlighted the need to simplify the overall 
narrative,  continue  efforts  to  gather  sponsors,  bring  forward  legacy  discussions  and 
business plans for flagship venues and clarify targets for international visibility.  

As  far  as  Leeuwarden  is  concerned,  at  the  end  of  the  monitoring  process,  the  panel 
praised the excellent work done in bringing the project forward, underlining in particular 
its strong European dimension, the high degree of involvement of local citizens and the 
focus  on  issues  of  global  and  European  reach  closely  linked  to  current  challenges.  It 
however  recommended  that  Leeuwarden  increase  the  project’s  international  visibility, 
including through cooperation with ad hoc national authorities, and involve all relevant 
stakeholders, from institutions to civil society at regional and local levels, in governance 
discussions in order to develop a sustainable legacy strategy. 

The monitoring process culminated with the panel making a positive recommendation to 
the  Commission  on  awarding  the  €1.5m  prize  in  honour  of  Melina  Mercouri  to  both 
cities  after  the  last  monitoring  meeting  in  April  2017.  Based  on  this  positive 

13  Council Decision No 2014/352/EU of 21 May 2014 designating the European Capital of Culture for the year 2018 

in the Netherlands (OJ L 175, 14.6.2014). 

7 

 
                                                                                                                                                                            
recommendation,  in  the  autumn  2017,  the  Commission  financed  the  prize  to  the  two 
ECOC 2018 from the budget of the EU Creative Europe programme14. 

The sub-sections below describe the main features of the ECOC programmes of Valletta 
and  Leeuwarden.  The  final  report  of  the  contract  supporting  the  evaluation  contains 
further  information,  including  the  way  the  programmes  had  a  European  dimension  and 
involved citizens (as these are the main two criteria of the ECOC Action) as well as on 
the development of the applications and the governance and funding structures. 

3.2.  Valletta 2018 

Valletta  is  the  capital  city  of  Malta  and  part  of  a  wider  metropolitan  area  of  nearly 
400,000  people,  representing  90%  of  the  country’s  population.  The  area  is  heavily 
dependent  on  leisure  tourism,  whilst  other  key  industries  include  medical  tourism, 
electronics, textiles and film production. The country’s history has given Valletta a rich 
cultural  and  architectural  heritage,  which  enabled  the  city  to  gain  UNESCO  World 
Heritage status in 1980. Valletta also has a long tradition in the fine arts. This led to the 
creation of the National Museum of Fine Arts, which operated from 1974 to 2016. The 
museum has now been replaced by a new institution, MUŻA (an acronym for “MUŻew 
Nazzjonali  tal-Arti”  in  Maltese),  which  had  a  partial  opening  towards  the  end  the  title-
year of Valletta 2018. Other notable aspects of Valletta’s cultural life include the Maltese 
Carnival as well as the Valletta International Baroque Festival. 

The initial overall slogan of Valletta 2018 was “Imagine 18”, which had emerged from 
the various consultation activities that had taken place during the application phase. The 
intention was to trigger positive responses supporting the ECOC project, inspire Maltese 
society  and  appeal  to  overseas  audiences  and  partners.  The  programme  was  articulated 
around  four  main  themes  (Generations,  Routes,  Cities  and  Islands)  and  pursued  four 
main  objectives:  Making  careers  of  culture  (supporting  the  development  of  artists  and 
creatives and growing the critical mass of audiences); Growing internationally (engaging 
across and beyond Europe’s borders);  Establishing Valletta as  a creative city  (targeting 
the  city’s  social,  economic  and  cultural  regeneration)  and  Nurturing  sustainable 
relationships  with  the  environment  (based  on  new  sustainable  connections  between 
citizens, architecture and landscape). In this way, Valletta 2018 was well connected with 
a  strategic  development  plan  for  Valletta  and  for  Malta.  It  also  had  the  unanimous 
support  of  all  68  local  Councils  in  the  country  and  cross-party  political  support  in 
Parliament. 

Early  in  the  development  phase,  the  initial  concept  of  “Imagine  2018”  was  however 
considered as being of limited relevance or usefulness and it was eventually dropped in 
favour of the strapline: “Valletta 2018: An island-wide fiesta”, reflecting the intention for 
the  ECOC  to  better  represent  the  whole  of  Malta  and  facilitate  the  participation  of 
citizens  across  the  national  territory.  Indeed,  this  theme  –  and  corresponding  thematic 
lines  (Island  Stories,  Future  Baroque  and  Voyages)  –  was  chosen  for  its  reference  to 
Malta’s local traditions and fiestas bringing whole communities together. At the time of 
the  final  monitoring  meeting,  Valletta  2018  reported,  however,  that  50  out  of  the  64 
projects included in the initial application were still underway. 

14   Regulation (EU) No 1295/2013 of the European Parliament and of the Council of 11 December 2013 establishing 
the  Creative  Europe  Programme  (2014  to  2020)  and  repealing  Decisions  No  1718/2006/EC,  No  1855/2006/EC 
and No 1041/2009/EC (OJ L 347, 20.12.2013, p. 221). 

8 

 
                                                           
The  responsibility  for  the  organization  and  implementation  of  the  ECOC  project  was 
entrusted  to  a  new  body,  the  Valletta  2018  Foundation,  which  had  been  created  to 
develop the application. The Foundation was led by a Board of Governors consisting of 
ten members, including the Mayor of the City of Valletta, the Rector of the University of 
Malta,  the  President  of  the  Malta  Chamber  of  Commerce,  Enterprise  and  Industry  and 
other  stakeholders.  The  Foundation  was  also  supported  by  a  wider  partnership  that 
included all  68  Local  Councils  (which each signed a charter), the Association  of  Local 
Council and Arts Council Malta. 

It is worth noting that during the development phase, the Foundation went through some 
considerable  changes  in  personnel,  including  at  senior  staff  level  in  2013,  and  again  in 
2017.  Moreover,  the  governance  of  the  ECOC  became  the  subject  of  much  political 
debate and disagreement. In particular, it attracted a considerable degree of national and 
international  criticism  following  comments  posted  on  social  media  by  the  Chairman, 
which appeared to mock the murdered journalist, Daphne Caruana Galizia15. 

As  far  as  budget  is  concerned,  the  figure  indicated  in  the  application  amounted  to 
€49.57m for the years 2013-20. The eventual operational budget was nevertheless much 
lower at €26.5m, €17.31m of which dedicated to the cultural programme itself. Most of 
the funding (€24.22m) came from the national Government. However, a positive point is 
that it seems a considerable proportion of this contribution consisted of new funds, while 
in the application a majority of the national Government’s contribution was supposed to 
come  from  reallocated  funds  as  opposed  to  new  funds  (€21.23m  out  of  a  total  of 
€39.67m). The budget for the cultural programme was also complemented by significant 
Government  investment  in  cultural  infrastructure.  Finally,  sponsorship  raised  €60k, 
which  was  much  less  than  originally  intended  (€2m).  The  EU  funding  provided  in  the 
form  of  the  Melina  Mercouri  Prize  was  added  to  the  general  budget  for  operating 
expenditure. 

3.3.  Leeuwarden 2018 

Located in the northwest of the Netherlands, Leeuwarden is a city of 100,000 inhabitants 
in  the  region  of  Friesland,  which  has  a  population  of  646,000  and  a  total  of  eleven 
historic cities connected together by water. Leeuwarden is also one of the oldest cities in 
the north of the Netherlands with a rich history dating back to the Roman age. Friesland 
has two official languages (Dutch and the local Frisian) and is the only bilingual region 
in  the  Netherlands  with  Frisian  being  taught  in  many  of  the  region’s  schools.  The 
inhabitants of Friesland are also recognised as having a strong cultural identity and sense 
of  local  pride.  The  city  and  the  region  are  relatively  well  served  as  far  as  cultural 
infrastructures (such as the impressive Fries Museum that opened in 2013) and activities  
(including  big  festivals)  are  concerned,  but  their  cultural  sector  is  often  seen  as  being 
locally-focussed  and  inward-looking  in  terms  of  cultural  content,  target  audience  and 
ambition. 

Leeuwarden associated the Friesland region in its bid, a key element of which was “Iepen 
(open) mienskip”. Mienskip is a Frisian word and tradition, associated with core values 
such  as  mutual  respect,  participation,  grass-roots  development,  equality  and  civic 
responsibility. The adjective “Iepen” adds an important second element of “openess” and 
expresses  the  need  for  the  city  and  the  region  to  welcome  external  input  and  become 
more  open-minded  and  outward-looking  in  including  those  not  belonging  originally  to 

15   https://lovinmalta.com/news/valletta-capital-of-culture-chairman-mocks-last-words-of-assassinated-journalist 

9 

 
                                                           
the “Frisian community”. The open mienskip approach was to dominate the development 
work  of  Leeuwarden-Friesland  2018,  as  well  as  the  content  of  its  cultural  programme, 
with a key overarching  principle being around using culture to help deal with society’s 
issues.  

Tackling  social  issues  was  indeed  a  key  part  of  the  bid-book  and  eventual  cultural 
programme of the ECOC-year, particularly in the fields of social inequality (such as child 
poverty),  ecology  (such  as  the  threat  to  biodiversity  or  water  management),  diversity 
(notably, minority manguages and the integration of migrants) as well as the relationship 
between the city and the countryside, which are challenges that are all common to many 
other cities and regions in Europe. Against this background, the cultural programme was 
organised around the following five thematic lines: Nature and Culture (using culture as a 
way to draw attention to nature); City and Countryside (exploring themes such as brain 
drain  from  countryside  to  cities  or  reflecting  on  how  to  create  creative  ecologies); 
Community and Diversity (promoting connections between people from different cultural 
backgrounds);  Lab  Lws  (using  culture  as  a  vehicle  for  innovation)  and  Royal  Frisians 
(highlighting  the  rich  culture  of  the  area).  More  specifically,  three  story  lines  were 
explored across these above overarching  themes: Dare to  Dream  (bringing the city  and 
region  to  the  European  stage),  Dare  to  be  Different  (opening  the  Frisian  community  to 
different cultures and Europe) and Dare to Act (protecting the unique biodiversity of the 
Frisian region). 

As with other ECOC, a dedicated delivery entity, the LF2018 Foundation, was set up as 
an independent body to specifically develop and deliver the programme.  It consisted of 
around  60  staff  members  linked  to  a  range  of  roles  including  marketing  and 
communication,  internationalisation,  production  as  well  as  finance.  The  Foundation’s 
Supervisory  Board  was  independent  of  the  region  and  city  authorities  and  consisted  of 
five  members  selected  from  different  perspectives  such  as  knowledge  areas  (artistic, 
legal, governance). The representatives of the region and city were connected through a 
regular formal meeting with the CEO and managing director of the LF2018. 

The  original  budget  foreseen  at  the  bid-book  stage  for  the  delivery  of  the  ECOC  was 
estimated at €74m, with the municipality of Leeuwarden, the other municipalities in the 
region and the province of Friesland contributing respectively €5.95m, €11m and €20m. 
The Dutch Ministry of Education, Culture and Science and the Dutch Ministry of Interior 
Affairs  pledged  to  contribute  €7.5m.  Expected  income  from  private  sponsors  was 
estimated at a comparatively high level, at €16m or 22% of the total budget. By the end 
of 2018, the actual budget  had increased to  €104.6m  (of which €84.4m  for the cultural 
programme),  meaning  spend  on  the  ECOC  increased  by  €30.3m  compared  to  the  bid-
book.  This  increase  in  budget  was  mainly  the  result  of  an  increased  income  from  the 
private sector (€50.8m instead of €22m) as ticket sales peaked at €20m instead of the €4 
million planned. 

4.  METHOD 

In order for results to be comparable with previous evaluations, the methodology for this 
evaluation  closely  followed  the  approach  adopted  in  previous  assessments  of  the 
Action16. The focus has been on research at city level and, in particular, the gathering of 
data  and  stakeholders'  views  from  both  Valletta  and  Leeuwarden.  The  main  evaluation 
sources can be identified as follows: 

16   See  previous  evaluation  reports  available  at:  https://ec.europa.eu/programmes/creative-europe/actions/capitals-

culture_en (table at the bottom of the document). 

10 

 
                                                           
-  EU level literature: this included higher level EU policy and legislative briefings, 
papers,  decisions  and  other  documents  relating  to  ECOC.  This  mainly  focussed 
on  reports  of  the  panels  for  selection  and  monitoring  and  the  original  bidding 
guidance  to  understand  how  the  two  ECOC  established  themselves  in  the  early 
days. Academic research was also consulted regarding the ECOC Action and the 
role of culture in the development of cities; 

-  ECOC  level  literature  from  Valletta  and  Leeuwarden-Friesland:  this  included 
original  bids  and  applications,  internal  reports  linked  to  the  application, 
monitoring or evaluation processes, as well as other pieces of literature analysing 
the cultural programme itself; 

- 

- 

-  Quantitative data: where available, evidence linked to each ECOC was collected 
in relation to budgets and expenditures, projects' numbers and types, participation 
and  audience  figures  as  well  as  other  pieces  of  quantitative  data  relevant  to  the 
assessment of the work and benefits of the ECOC in each city; 
Interviews with managing teams: those responsible for the day-to-day design and 
delivery of the ECOC were interviewed in each city during city visits in late 2018 
(i.e. during the host  year) and in  spring 2019 (13 interviews of the  Leeuwarden 
team and 10 interviews for the Valletta team). Almost all of the key individuals 
within the delivery agencies were interviewed including those linked to strategic 
development,  marketing  and  communication,  project 
implementation  and 
financial management; 
Interviews  with  other  key  stakeholders17:  mainly  face-to-face  interviews  were 
undertaken  with  stakeholders  both  directly  and  indirectly  involved  in  either  the 
planning  or  delivery  of  the  ECOC  along  with  those  more  widely  linked  to  the 
cultural,  social,  economic  or  political  agenda  of  the  host  cities.  Stakeholders 
included 
in  cultural  organisations,  city/regional/national 
administrations,  tourism  and  visitor  agencies,  media  organisations  as  well  as 
voluntary  and  community  organisations.  Interviews  were  also  conducted  with 
managers  of  individual  projects  and  activities  supported  through  the  ECOC 
Action  that  made  up  the  cultural  programme  of  each  city.  A  number  of 
international  artists  and  cultural  operators  were  interviewed,  which  provided  an 
external perspective on the ECOC; 
Interviews with EU-level stakeholders: two former members of the selection and 
monitoring  panel  gave  feedback  while  EU-level  cultural  bodies  were  invited  to 
contribute but only one responded; 

those  working 

- 

-  The evaluation does not include a wider public consultation. As explained in the 
roadmap18, the Action is considered to be local while international participation is 
scattered within and outside Europe and is difficult to reach. On the other hand, 
the opinions would be based on attendance to specific events and would not give 
useful insights for the evaluation of the ECOC Action as a whole.  

The  final  report  of  the  contract  supporting  the  evaluation  provides  a  detailed 
understanding of the 2018 ECOC Action and within this an assessment of the work and 
progress  of  Valletta  and  Leeuwarden.  There  are,  however,  issues  to  consider  when 
assessing the strengths of the evidence base used for this study: 

17   On  top  of  interviews  with  members  of  the  managing  teams  of  the  two  ECOC  2018,  17  stakeholders  for 

Leeuwarden and 30 stakeholders for Valletta were interviewed. 

18   See Ares (2018) 1933090 of 11/4/2018. 

11 

 
                                                           
-  Although both cities have undertaken some form of evaluation work themselves, 
part  of  the  results  of  these  studies  were  not  yet  available  at  the  time  of  the 
evaluation.  The  European  evaluation  of  the  ECOC  Action  has  used  as  much  of 
this secondary information as possible, but could not benefit from its final results, 
especially for what concerns quantitative data; 

-  As  already  mentioned  in  previous  evaluations,  an  ideal  way  of  conducting  this 
evaluation would entail a before ('baseline') study and an after-picture ('ex-post') 
study, instead of carrying out the latter alone; the impossibility of comparing the 
two studies affects the accuracy of the evaluation. Moreover, since the evaluation 
was  undertaken  during  and  shortly  after  the  end  of  the  ECOC  time  framework 
some  of  the  effects  of  the  programme  had  not  manifested  themselves  entirely. 
Many stakeholders involved in the evaluation commented that the real impact of 
the ECOC on the city and its residents would take time to filter through. In this 
respect,  the  Commission  once  again  highlights  that  budget19  and  timing20  only 
allow an ex-post evaluation to take place and therefore only an after picture has 
been studied; 

-  An ulterior consequence of the modest yearly budget allocated to the evaluation is 
the fact that the primary evidence data gathering tends to be more of qualitative 
than  quantitative  nature;  while  qualitative  data  still  holds  a  great  importance  in 
the  evaluation,  the  lack  of  diversity  of  data  sources  translates  into  a  lesser 
dependability, for instance, in the process of proving the objective outcomes and 
impacts of ECOC on widening participation in culture; 

-  Detailed  modelling,  economic  impact  assessments  or  large-scale  surveys  were 
outside  the  scope  of  this  study.  This  evidence  used  outside  of  the  interviews  is 
dependent on the local evaluation and other research commissioned by the ECOC 
cities.  Both  2018  ECOC  cities  did  not  commission  large  and  ambitious 
evaluations providing quantitative data or researching the economic impact of the 
programme  or  levels  of  cultural  awareness  before,  during  and  after  the  ECOC 
year. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

This part of the report presents a diagnosis of the implementation of the ECOC Action in 
2018. It draws together the results relating to the two ECOC 2018 and to a lesser extent 
the findings from previous ECOC evaluations. 

All figures appearing in the following sub-sections come directly from the report of the 
external ex-post evaluation of the two ECOC 2018. The reader will find in the full text of 
the document many more examples illustrating the conclusions presented below. 

Very often, examples and figures can illustrate different points made under "relevance", 
"efficiency",  "effectiveness",  "sustainability",  "EU  added  value"  and  "coherence".  They 
are  therefore  sometimes  repeated  in  various  sub-sections.  However,  for  the  ease  of  the 
reading,  repetitions  have  been  limited  to  the  minimum  and  the  reader  is  invited  to 
correlate some conclusions presented below with examples or figures that may have been 
given in previous paragraphs or may be given in following ones. 

19   The budget allocated to the evaluation work (75 000 €) is proportionate to the low level of EU funding directly 
provided to the ECOC (i.e. an award based on a recommendation of the panel after the final monitoring meeting 
in the form of the €1.5m Melina Mercouri Prize). 

20   Decision No 1622/2006/EC requires that the Commission conduct the evaluation immediately after the title year. 

12 

 
                                                           
5.1.  Relevance 

Findings from the ECOC 2018 evaluation show that the two title-holders developed and 
implemented cultural programmes that were consistent with the EU Treaty, in particular 
Article 16721, as well as with Decision No 1622/2006/EC. Indeed, the ECOC Action was 
a  good  vehicle  for  the  two  hosting  cities  in  2018  to  promote  cultural  diversity  and 
highlight  common  elements,  contributing  in  this  way  to  the  "flowering"  of  the  local, 
national  and  European  cultural  scene.  As  is  shown  in  following  sub-sections  through 
many  concrete  examples,  the  two  cities  organized  thousands  of  events  on  top  of  their 
usual  cultural  offer.  These  events  covered  all  types  of  cultural  and  artistic  disciplines, 
targeted  many  groups  of  audiences  and  presented  various  aspects  of  the  diversity  and 
common features of cultures in Europe, notably through co-operations with partners from 
other European countries and beyond. 

It is worth noting that despite considerable changes in the artistic themes, governance and 
other aspects of Valletta 2018 during the development phase (as mentioned before), the 
objectives of the revised programme remained broadly consistent with the spirit of three 
of the four original objectives set in the original application (with less focus being given 
to  the  fourth  relating  to  the  environment)  and  thus  with  the  objectives  set  at  EU  level. 
This is illustrated in the sections below. 

The European dimension in Valletta 

At  the  time  of  the  selection,  the  panel  did  not  consider  that  Valletta  2018  had  a 
particularly  strong  European  dimension.  During  the  development  phase,  the  panel  thus 
made recommendations to help the ECOC delivery Foundation improve it, in particular 
through  the  themes  put  forward  in  individual  projects,  intensified  co-operation  with 
European  artists  and  cultural  operators  and  increased  efforts  to  make  Maltese  citizens 
more aware of Europe  and European  citizens more aware of Malta.  In reaction to  such 
recommendations, the Foundation took specific measures, in particular the appointment 
of  an  International  Artistic  Advisor,  who  proved  very  instrumental  in  restructuring  the 
programme around a stronger international dimension and in connecting the Foundation 
and Malta’s cultural stakeholders to international artists. 

The view of the evaluator is that by the end of the ECOC-year, the cultural programme of 
Valletta  2018  featured  more  activities  promoting  the  cultural  diversity  of  Europe  and 
highlighting the common aspects of European cultures than Malta’s cultural offering in 
previous  years. Explicit objectives were set relating to the European (and international) 
dimension,  such  as  integrating  Malta’s  rapidly-changing  and  diverse  society  into  the 
programme,  as  well  as  engaging  communities  across  and  beyond  Europe’s  borders.  A 
good  illustration  is  the  development  of  projects  exploring  the  theme  of  European 
migration  as  part  of  a  multi-project  strand  (added  during  the  preparatory  phase  as  a 
response to the panel’s request), “Exile & Conflict”, which featured several projects, or 
the production of new opera entirely in Maltese, “Aħna Refuġjati”, addressing the very 
topical issue of large-scale migration across the Mediterranean. 

There were also more collaborations with European artists and operators from different 
countries.  New  international  partnerships  were  established  and  the  cultural  programme 
featured  more  performances  by  international  artists  and  exhibitions  of  international 
works  and  residencies  by  European  artists.  According  to  research  carried  out  locally, 
over 200 international artists were to be part of the programme, from  countries such as 

21   http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A12008E167.  

13 

 
                                                           
Egypt, Tunisia, Syria, Jordan and Lebanon (to reflect the Mediterranean dimension of the 
ECOC), as well as Europe and the rest of the world. As an example, the “Meeting Point” 
strand  (in  charge  of  connecting  projects,  people,  disciplines  and  genres)  arranged  eight 
residency  programmes  across  Malta,  which  enabled  50  European  artists  to  complete 
residencies  in  the  country,  as  well  as  three  outgoing  residencies  and  three  international 
workshops.  However,  several  international  artists  responsible  for  key  projects  reported 
that  their  overall  experience  of  Valletta  2018  had  been  somewhat  negative  due  to 
challenges  in  governance  (see  above).  Similarly,  the  political  controversies  associated 
with  Valletta  2018  had  the  effect  of  attracting  negative  international  attention  and 
damping the enthusiasm of some international partners. 

Finally, the tourism data (particularly the increased cultural motivation to visit Valletta) 
suggests some success in reaching a European public (see below under “effectiveness in 
reaching out European and international audiences”).  

The “City and Citizens” dimension in Valletta 

In  practice,  the  main  effort  was  put  into  connecting  the  ECOC  with  citizens  and 
audiences  across Malta. This was a key element of Valletta 2018’s application and this 
remained a strong focus of the Foundation’s work all along the development phase of the 
project. The initial idea was that the ECOC project would encompass the whole territory 
of the Maltese islands, and to do that, it was decided to recruit a team of 6 regional co-
ordinators,  with  the  responsibility  to  undertake  community  outreach  for  different 
territories. In the evaluator’s view, this proved an interesting element in the Foundation’s 
governance. 

On  top  of  that,  one  of  the  sub-sections  of  the  Valletta  2018  programme  featured  13 
projects specifically focussed on involving children and young people from across Malta, 
as these were supposed to be primary targets of the ECOC. In particular, work was done 
directly with schools, with a dedicated programme that involved around 28,500 children 
in creative projects over the period 2016-201822.  

Finally, there was some activity, albeit relatively limited, to connect with and involve the 
Maltese  diaspora  internationally,  such  as  a  site-specific  installation  and  video  work, 
bringing  the  observations  and  recollections  of  the  Maltese  diaspora  to  an  audience  in 
Malta. 

The European dimension in Leeuwarden 

According  to  the  evaluator,  Leeuwarden-Friesland  2018  took  the  European  dimension 
very  seriously.  An  extremely  important  point  in  this  regard  is  that  while  other  ECOC 
tend to ensure that staff in general consider the European dimension with no overall lead 
in charge of pro-actively pushing this agenda forward, a key member of the LF2018 team 
was  specifically  responsible  for  promoting  and  embedding  the  European  dimension  at 
both  programme  and  project  levels.  The  responsibility  of  this  person  was  to  check, 
monitor and encourage  project  owners, the ECOC Board and fellow management team 
members to promote a European aspect to their daily work and thinking.  

As  a  result  of  this  strong  commitment,  there  were  overall  1,600  international 
collaborations  covering  87  countries23,  which  was  five  times  greater  than  originally 
planned.  An  example  is  the  “11  Fountains”  project  –  11  artists  from  across  the  world 

22   Valletta 2018 progress report for the second monitoring meeting. 

23   Research undertaken by LF2018. 

14 

 
                                                           
were commissioned to design and build 11 new fountains with local residents in each of 
the  11  Frisian  cities  to  promote  the  importance  of  water  in  the  region.  Another  telling 
illustration  is  the  “Potatoes  go  wild”  project,  which  consisted  in  a  series  of  cultural 
activities  that  involved  artists  from  Malta,  Germany,  Ireland  and  Serbia  using  the 
common theme of the potato to share European stories.  

The ECOC programme  also  addressed  a number of key “European themes” that linked 
Leeuwarden-Friesland  with  other  regions  across  the  EU  Member  States  facing  similar 
challenges  and issues  (e.g. minority languages,  migration, urban/rural  pressures  as  well 
as  a  range  of  environment  and  ecological  issues).  A  good  example  is  the  “Lost  in  the 
greenhouse” project,  which focused on migration in Europe and included performances 
produced with local people and migrants living in the area, especially Polish workers. It 
covered a range of European issues including  disparities in  wage levels  across the EU, 
skilled labour moving to Western Europe, integration issues with newly arrived migrants 
as well as the economic consequences and benefits of migration in different countries. 

Another  strong  aspect  of  Leeuwarden-Friesland’s  approach  to  the  European  dimension 
was the existence of funding to  make it easier for projects  to  find partners from  across 
Europe. Some €265k from the open programme (i.e. the part of the cultural programme 
that  dealt  with  the  bottom-up  mienskip  approach)  was  ring-fenced  to  help  smaller 
organisations find European partners, for example, by covering the costs of visits. 

The ECOC in  Leeuwarden-Friesland also had a number of processes in  place to  check, 
report and monitor the extent to  which the programme and its projects  had a European 
dimension.  According  to  the  evaluator,  this  was  instrumental  in  ensuring  that  this 
dimension was not lost along the way. 

The “City and Citizens” dimension in Leeuwarden 

It  is  the  view  of  the  evaluator  that  by  far  the  most  unique  and  key  principle  of  the 
Leeuwarden-Friesland 2018 was linked to the “City and Citizens” dimension, reflecting 
the ECOC mienskip concept.  

Engaging with and encouraging local people to develop and implement  projects for the 
cultural programme was a key defining aspect of the ECOC and something that those in 
the  LF2018  Foundation  were  totally  committed  to  delivering.  The  final  report  of  the 
contract supporting the evaluation concludes that this engagement attitude was the central 
spine of the whole project  rather than just a specific and separate community  ‘add-on’, 
which  was  sometimes  the  case  with  past  ECOC.  Critically,  those  responsible  for 
overseeing the ECOC in the Foundation saw the citizens of Leeuwarden-Friesland as the 
producers  of  the  ECOC  rather  than  simply  the  consumers  of  it  (i.e.  were  part  of  the 
ECOC’s delivery and not just a passive audience). More than 700 of the 800 projects in 
the cultural programme came from the mienskip approach, where the communities came 
together to design, plan, often fund and deliver various cultural projects. Three dedicated 
staff  members  in  the  LF2018  Foundation  were  in  place  to  help  develop  this  open 
programme, another example on how governance was aligned on declared intentions.  

Another key impact was that the ECOC stimulated a new wave of amateur performers in 
the region, who wanted to directly produce rather than just consume culture. Professional 
artists, performers and technical staff still played a key part in the programme, but a key 
impact  was  awakening  and  stimulating  non-professionals  to  consider  performing  and 
participating in culture. Over 60,000 local people took part as performers or volunteers, 

15 

 
including  teachers,  farmers,  office  workers,  restaurant  owners,  people  out  of  work  and 
schoolchildren as well as cultural professionals24. 

5.2.  Efficiency 

As was already the case in previous ex post evaluations, it also appears this year from the 
report of the contractor that overall, the ECOC remains an efficient EU Action, providing 
good  levels  of  benefits  for  the  EU  for  relatively  little  EU  investment.  Indeed,  the  only 
direct contribution from the European Union, in the form of the €1,5m Melina Mercouri 
prize awarded to each ECOC, is dwarfed by the total amount of money invested by the 
host  cities  in  designing  and  delivering  the  ECOC  (the  operational  budgets  were 
approximately €104.6m for Leeuwarden and € 26.5m for Valletta). 

Without  the  initial  EU  impetus  to  support  the  ECOC,  it  is  unlikely  that  the  host  cities 
would have invested anywhere near the amount of funding they did in  connection with 
the ECOC title. The possibility of securing the title typically stimulates cities  – but also 
their  respective  regional  /  national  public  authorities  and  private  partners  –  to  invest 
much more heavily in culture than they would in the absence of the ECOC. This is true 
both  in  terms  of infrastructure  (this  was more the case in  Valletta than in  Leeuwarden, 
and  that  is  precisely  why  the  competition  started  one  year  ahead  of  schedule  in  Malta) 
and  expenditure  in  cultural  events  and  activities  (especially  in  Leeuwarden).  The  final 
report  of  the  contract  supporting  the  evaluation  concludes  that  this  means  the  Action 
remains highly efficient in terms of returns from the Melina Mercouri prize.  

As described above, the efficiency of management arrangements varied between the two 
cities, with those in Leeuwarden-Friesland being generally stronger than in Valletta. 

Indeed Valletta went through considerable changes and lost important expertise en route, 
which  ultimately  had  an  impact  on  the  artistic  direction  and  content  of  the  cultural 
programme.  Whilst  opinions  were  divergent  or  even    polarised,  many  key  stakeholders 
(including  those  involved  in  implementing  the  programme  and  those  external  to  the 
implementation)  expressed  the  view  that  the  overall  artistic  direction  suffered  from  the 
above-mentioned  departure  of  key  staff  members  of  the  Valletta  2018  Foundation  in 
2013  and  then  again  in  2017.  The  political  dimension  of  the  Foundation  also  attracted 
criticism locally and internationally. Furthermore, as indicated above, funding in Valletta 
was considerably lower than the one planned in the application, even though it was still 
sufficient  to  produce  a  programme  of  the  scope  and  scale  of  a  European  Capital  of 
Culture as illustrated below. 

In  Leeuwarden,  the  arrangements  for  governance  and  implementation  proved  efficient, 
drawing  on  the  strong  political  support  in  the  city,  the  region  and  the  province. 
According  to  the  evaluator,  the  ECOC  was  well  run  with  strong  and  robust  systems, 
procedures,  transparency  and  leadership.  This  is  reflected  firstly  in  the  fact  that  the 
Foundation and its partners delivered a cultural programme of the intended scope, scale 
and  quality  and  raised  the  necessary  resources  (the  final  resources  income  were  even 
much  higher  than  planned  as  described  above).  This  conclusion  finds  another 
confirmation in that the Foundation generally fulfilled the expectations set for the ECOC, 
notably  in  terms  of  audience  engagement,  which  was  at  the  very  core  of  the  mienskip 
concept.  In particular, it appears that there was a genuine link between the ECOC staff 
and local people, with a high level of trust and openness between the two. This trust and 

24   Figure supplied by the independent Working  Group on Monitoring and Evaluation of the province of Friesland 

and the municipality of Leeuwarden. 

16 

 
                                                           
openness manifested itself in relation to an ECOC programme that had over 800 projects 
spread across a series of social themes and a wide geographic area. 

The  EU-level  selection  procedure  ensured  the  designation  of  two  ECOC  that  were 
relevant to the objectives at EU level and that had the potential to achieve the intended 
effects.  The  procedure  ensured  healthy  competition  in  the  Netherlands  with  five  cities 
submitting strong applications. In Malta, the procedure provided sufficiently rigorous and 
impartial scrutiny of the Valletta’s ECOC application. However, the Maltese experience 
suggests  that  the  procedures  introduced  by  Decision  1622/2006/EC  might  not  be 
appropriate  to  a  situation  in  which  the  national  Government  not  only  organises  the 
selection  process  but  is  also  –  by  necessity  –  one  of  the  key  players  in  an  ECOC 
application (even if indirectly, e.g. via a body under its control). It is worth noticing that 
in  the  meantime,  the  selection  procedure  has  been  changed  by  Decision  445/2014/EU, 
providing that the panel is now composed of 10 members designated by EU institutions 
and bodies (against 7 before) and up to 2 by national authorities (as against 6 before). 

The  EU-level  monitoring  procedure,  with  formal  and  informal  monitoring  meetings  as 
well as in situ visits with the panel, has proven valuable in  giving impartial advice and 
support to the ECOC from highly experienced experts, some of whom have implemented 
previous ECOC.  

5.3.  Effectiveness 

Ultimately, the effectiveness of the ECOC in achieving the objectives set for them (at EU 
level and local level) can only be determined in the long-run. That being said, the final 
report of the contract supporting the evaluation shows that the two ECOC 2018 presented 
cultural  programmes  that  were  more  extensive,  diverse,  innovative  and  international 
compared to the baseline cultural offering in previous years.  

In  total,  the  ECOC  cultural  programme  in  Leeuwarden-Friesland  was  ambitious  in  its 
scale  and  focus,  with  accent  put  on  the  social  dimension.  It  included  800  projects  of 
various sizes and formats, representing a whole range of cultural disciplines, with most of 
them  having  a  strong  social  component.  As  already  mentioned,  and  in  line  with  the 
mienskip  approach,  700  of  these  projects  were  bottom-up  and  emerged  from  local 
people, some of them implementing a cultural project for the first time in their life. One 
can therefore conclude that an overwhelming majority of the projects in the final ECOC 
programme were genuinely additional. In Valletta, the programme featured 168 projects 
involving  nearly  500  events.  Whilst  some  parts  of  the  programme  represented  a 
continuation of existing events (e.g. some festivals), some important projects were new, 
notably big events  such  as the opening ceremony, the Valletta Pageant  of the Seas, the 
closing  ceremony  or  key  events  curated  by  international  artists.  A  higher  number  of 
European  and  other  international  artists  were  involved  in  both  ECOC  compared  to 
previous years (see paragraphs on the European dimension above), some new works were 
created and performed or exhibited and spaces in both cities were used in new ways to 
host cultural events. 

As  described  below,  both  ECOC  helped  widen  access  to  and  participation  in  culture 
although  the  evidence  is  stronger  in  Leeuwarden  than  in  Valletta.  Both  ECOC  also 
helped  strengthen  the  cultural  capacity  of  their  cultural  and  creative  sectors  as  well  as 
their links with other sectors. Finally, both ECOC widened access to and participation in 
culture  during  2018,  although  Valletta  suffered  considerable  adverse  international 
publicity.  

Effectiveness in delivering a cultural programme with a European dimension 

17 

 
The Leeuwarden-Friesland cultural programme was of the scale and quality proposed in 
the  original  application  and  had  a  genuine  European  dimension.  According  to  the 
evaluation, the ECOC project has been successful in contributing to making the cultural 
offering  of  Leeuwarden-Friesland  more  European.  Indeed,  as  already  illustrated  above, 
the cultural programme featured performances by a diversity of international artists and 
exhibitions  of  international  works,  strengthened  European  networks  and  connections, 
European co-productions and residencies by European artists.  

The performance of Valletta was also relatively good in this respect. It appears that the 
vast majority (92%) of respondents responsible for individual projects viewed their work 
as relevant to European audiences, while 69% believed their project could be exported in 
the future25. Reflecting the European dimension, data from  a large-scale cross-sectional 
survey  undertaken  within  the  Valletta  2018  research  programme  also  shows  that  the 
sense  of  attachment  to  Europe  amongst  residents  of  Malta  increased  significantly,  with 
73% feeling “very attached” in 2018 compared to 34% in 201626. 

Effectiveness  in  reaching and engaging  with local  citizens, in  targeting  specific groups 
and in strengthening the local cultural and creative sectors 

Both  ECOC  widened  access  to  and  participation  in  culture  during  2018,  although  the 
evidence is stronger in Leeuwarden than in Valletta.  

In  Leeuwarden-Friesland,  audience  figures  from  the  local  research  showed  that  5.4 
million  people  attended  ECOC  projects  up  to  and  including  2018,  a  figure  which 
increased by 5% each year from 2015 onwards. Some 51% of the audiences were people 
living  in  Friesland  with  a  further  42%  coming  from  the  rest  of  the  Netherlands. 
Significant parts of the local population participated in or attended the larger events, with 
71% of all 18-84 years olds attending the flagship parade of Royal de Luxe Giants and 
32%  of  this  age  group  attending  the  Escher  exhibition,  which  was  of  the  flagship 
exhibitions of the year. Local research showed that 30% of people attending events had 
little previous affinity to culture meaning a broader range of participation was achieved 
over the year. 

Perhaps equally impressive is that around 10% of the total population of the entire region 
(around 62,000 people) actually participated in ECOC activities either in terms of being a 
volunteer27 or delivering an ECOC project (i.e. was a producer and not just a consumer). 
Furthermore, according to a study published by the Friesland Cultural Planning Bureau, 
68% of the total  population  of Friesland felt involved in  the ECOC and concluded that 
access levels into the programme were high. 

The ECOC enabled Valletta to attract much greater audiences for culture than in previous 
years,  exceeding  400,000  people  for  the  whole  of  2018.  More  than  half  of  the  total 
audience figure was accounted for by events that were new for 2018, with the big open-
air  events  accounting  for  much  of  this  figure.  Valletta  2018  contributed  to  increased 

25   The Impacts of the European Capital of Culture Valletta 2018: Final Research Report, p.39. 

26   The Impacts of the European Capital of Culture Valletta 2018: Final Research Report, p.50. 

27   According to a research from the Foundation, 16,159 people registered as volunteers. Approximately 1,500 were 
‘core’ volunteers who volunteered on a regular basis. However, the overall number of volunteers was likely to be 
much  higher  than  the  number  of  people  actually  registered  because  project  monitoring  forms  (which  requested 
each  project  to  count  the  volunteers  they  used)  suggested  larger  volunteer  figures.  As  an  example,  monitoring 
figures  from  the  “Under  the  Tower”  project  alone  showed  it  used  14,595  volunteers  over  the  ECOC  year  on  a 
variety of roles, including promotion, sign-posting and setting up and taking down stages. 

18 

 
                                                           
attendance at or participation in cultural events amongst Maltese residents and provided 
new opportunities for citizens to be involved as creators, performers and audiences, for 
example by working with international artists in productions of high professional quality. 
At the same time, the total audience was considerably less than the stated aspiration of 1 
million. 

Finally, Valletta 2018 run an ambitious volunteer programme, Tal-Kultura, which began 
in 2016. Aggregate data on the total number of volunteers by the end of the title-year was 
not available at the time of the evaluation. However, before the start of the title-year, the 
programme  had  already  400  volunteers  registered  (equalling  2,500  hours  of  volunteer 
time), varying in age, nationality and background, and with tasks ranging from ushering 
and guiding to archiving and documentation.  

Both ECOC also helped strengthen the cultural capacity of the local cultural and creative 
sectors  and  their  links  with  other  sectors.  As  far  as  Valletta  is  concerned,  the  ECOC 
contributed  to  its  culture-driven  regeneration  by  serving  as  a  focus  for  cultural 
infrastructure  investments,  helping  to  stimulate  private  investments  and  implementing 
cultural activities that attracted visitors and increased the city’s vibrancy. Valletta 2018 
also  helped  develop  the  capacity  of  Malta’s  cultural  and  creative  sectors,  not  least 
through  the  involvement  of  some  645  cultural  operators  in  the  cultural  programme.  At 
the  same  time,  it  is  the  view  of  the  evaluator  that  the  experience  of  2018  highlighted 
some  gaps  and  therefore  the  need  to  develop  further  Malta’s  cultural  infrastructure  as 
well as reinforce capacity building, knowledge, competencies and skills amongst cultural 
players. Furthermore, some international artists established relationships with local artists 
that  will  endure  beyond  2018,  although  the  governance  challenges  and  political 
controversies tended to reduce the willingness of international artists to work in Malta in 
future.  

In Leeuwarden-Friesland, capacity was built both in terms of strengthening the skills and 
abilities  of  cultural  organisations  already  established  in  the  area  and  in  relation  to  the 
‘new’  cultural  players  who  were  involved  in  the  open  programme.  Helping  build  the 
capacity  of  grass  roots  organisations  or  individuals  who  often  possessed  very  little 
cultural  skills  prior  to  2018  was  particularly  driven  by  the  1.28  million  euros  that  was 
spent on project support in the open programme. The LF2018 Foundation also developed 
an online toolkit specifically to help people develop projects by providing a step by step 
guide  on  issues  including  project  design,  funding  availability,  partner  development, 
health  and  safety,  marketing,  securing  permits  finding  venues  and  maximising  ticket 
sales. The Foundation also organised six weekly meetings with all projects to bring them 
together in order to share news, requirements but also to share knowledge, good practice 
and advice. 

Effectiveness in reaching out to European and international audiences  

According to the evaluation, the ECOC raised the international profile of both 2018 title-
holding cities. 

As  far  as  Leeuwarden-Friesland  is  concerned,  many  stakeholders  interviewed  for  this 
evaluation  stated  that  it  would  have  been  far  easier  to  produce  a  more  “conventional” 
cultural programme with much less community participation and a larger number of “box 
office” projects that would have easily attracted the attention of international audiences. 
Against  this  background,  one  can  see  the  fact  that  some  7%  of  the  audiences  (around 
300,000 people)28 were international as a success indicator, even more so as the region 

28   As a comparison, the average of European and international audiences for previous ECOC is at around 5%. 

19 

 
                                                           
was  not  on  the  typical  tourist  route  for  many  international  visitors  to  the  Netherlands 
before  the  ECOC-year.  Overnight  stays  (often  seen  as  a  good  proxy  for  international 
visitors) showed an increase from 0.8 million in 2017 to 2.1 million in 2018. Estimates 
from  the  hotel  managers  suggested  around  65%  of  the  ‘extra’  overnight  stays  seen  in 
2018 compared to  2017  were from  international  visitors.  It  is  therefore safe to say that 
the  ECOC  raised  the  international  profile  of  Leeuwarden-Friesland,  albeit  from  a  low 
base.  

The  evidence  suggests  that  the  ECOC  contributed  to  an  increase  in  foreign  tourists  to 
Valletta. Tourist visits to Valetta increased from 1.61m in 2015 to 1.75m in 2016, 2.04m 
in 2017 and 2.37 in 2018. Of course, the overall trend of recent years is one of increased 
tourist  visits  to  the  country  in  general  (+12.7%  in  air  passenger  numbers  from  2017  to 
2018)  and  it  is  impossible  to  isolate  the  impact  of  the  ECOC.  However,  more  tourists 
reported  that  they  visited  Valletta  for  cultural  reasons  in  2018  (475,000)  than  in  2017 
(139,000)29. In this context, it is safe to conclude that the overall Government strategy (of 
which the ECOC constituted an important part) to regenerate and promote Valletta as a 
cultural destination is proving successful.  

Effectiveness in ensuring sustainability 

The  two  2018  ECOC  cities  offer  somewhat  different  potential  for  the  sustainability  of 
their activities and of improved cultural governance.  

The legacy plans for Leeuwarden-Friesland were relatively less developed compared to 
other  aspects  of  delivery  at  the  end  of  their  ECOC  year.  There  was  no  continuous  and 
smooth transition into 2019 with a clear and immediate published plan for new cultural 
activity for the region. However, clearer and stronger legacy plans were announced part 
way  through  2019,  which  gives  clearer  direction  in  terms  of  legacy  and  succession 
planning. The legacy issues relate more to people rather than bricks and mortar in that the 
ECOC  was  about  getting  people  to  participate  in  society  through  culture  rather  than 
building  new  cultural  facilities  and  venues.  This  means  the  sustainability  of  the  effects 
are  partly  up  to  whether  local  people  continue  to  be  engaged  and  whether  any  legacy 
organisation post-LF2018 continues  to  stimulate a bottom-up approach. Although some 
funding is in place and capacity has been greatly built up in 2018, there is still a concern 
that  momentum  will  be  lost  and  will  not  automatically  spring  into  life  in  the  coming 
years. 

It is the view of the evaluator that many of the positive effects of Valletta 2018 will be 
sustained  and  enhanced  beyond  2018.  The  Maltese  Government  continues  to  prioritise 
culture and the culture-driven regeneration of Valletta, as evidenced by the 19% increase 
in its budget for culture in 2019 compared to 2018. A key factor has been the creation of 
the  Valletta  Cultural  Agency  “to  carry  forward  the  legacy  created  by  Valletta  2018”, 
which will employ several staff members of the Foundation and operate from the same 
premises at the centre of Valletta.  

5.4.  Coherence  

The ECOC Action is coherent and complementary to the Creative Europe Programme in 
that it promotes the objectives of Creative Europe and is distinct from the other activities 
supported  by  the  programme.  Leeuwarden  and  Valletta  also  made  use  of  funding  from 
Creative  Europe  to  support  projects  in  their  cultural  programmes.  Examples  include 

29   MTA Heritage Locality Survey. 

20 

 
                                                           
“Orfeo  &  Majnun”,  which  was  an  interdisciplinary,  participatory  music-theatre  project 
featuring  partners  from  Malta,  Austria,  Belgium,  France,  Netherlands,  Poland  and 
Portugal. 

The ECOC Action is also coherent with and complementary to the European Structural 
and Investment Funds, depending on the context of each city holding the title. This was 
most notably the case in Valletta, where some significant investments were implemented 
in cultural infrastructure receiving co-financing from the ERDF, such as the new MUŻA 
(which opened the end of the ECOC-year) and the new Valletta Design Cluster (which is 
a  key  element  of  the  year’s  legacy).  Those  investments  thus  supported  the  successful 
implementation  of  the  ECOC  cultural  programme,  whilst  the  ECOC  helped  “valorise” 
the  investments  by  ensuring  that  the  venues  were  used  and  given  greater  profile.  In 
Valletta, Interreg also supported Design4Innovation, a collaborative project organised by 
eight  European  partners  dedicated  to  promoting  design  as  a  tool  for  user-  centred 
innovation. 

In  line  with  the  aims  of  the  2018  European  Year  of  Cultural  Heritage  (EYCH),  the 
programmes  of  both  Valletta  and  Leeuwarden-Friesland  included  projects  promoting 
cultural  tangible  and  intangible  heritage.  As  an  example,  five  projects  that  had  been 
awarded the EYCH 2018 label were presented in the “European Project Slam” during the 
annual conference of the Network of European Museum Organisations, which was held 
in Valletta during the title-year.  

5.5.  EU added value 

As  illustrated  above,  the  ECOC  Action  achieved  an  impact  that  would  not  have  arisen 
through the actions of Member States alone. 

The designation of Leeuwarden and Valletta as ECOC attracted benefits that would have 
been  unlikely  to  arise  to  the  same  extent  in  the  absence  of  the  Action.  Indeed,  the 
evidence presented in the chapters above suggest that the ECOC, as an EU initiative and 
an EU branding, provides the stimulus for stakeholders to commit resources and effort to 
a  shared  vision  and  collaborative  programme  at  a  much  greater  scale  than  would 
otherwise happen. The EU value added consists largely in the force of the 'brand' itself to 
act as a significant generator of interest from stakeholders not only from the city and but 
also from far beyond. 

This is evidenced, for example, by the creation of dedicated organisations to implement 
the ECOC and the allocation of resources that are additional to the “mainstream” funding 
for cultural activities in the cities and regions concerned. 

For Leeuwarden-Friesland, it is the view of the evaluator that the EU added-value is also 
to be seen in relation to  the ECOC’s very special profile, in that Leeuwarden-Friesland 
was  successful  in  stimulating  over  60,000  local  people  to  become  involved  in  the 
delivery  of  the  programme  in  a  way  that  no  national  or  local  ‘brand’  never  achieved 
before. Another element of this value added is connected to the need to have a European 
dimension,  which  many  saw  as  a  strong  driver  to  force  the  (cultural)  community  of 
Friesland to become more outward-looking and move away from its local comfort zones.  

In Valletta, the award of the title provided impetus to the wider culture-driven generation 
of the city, whilst the fixed timescale of the title-year provided greater focus than might 
otherwise  have  been  the  case  to  move  forward  in  that  direction.  While  the  national 
Government’s commitment to culture means that an increase in investment in venues and 
events  may  have  taken  place  anyway,  the  opportunity  of  the  title-year  led  to  those 

21 

 
investments  being  of  greater  scale  and  ambition  than  would  otherwise  have  been  the 
case. 

Regarding the visibility of the EU, both title-holders gave full prominence to the title of 
“European  Capital  of  Culture”  in  their  communication  and  promotional  materials  and 
gave prominence to the fact that the ECOC is an EU Action.  

The  results  consistently  achieved  by  the  Action  through  the  years  establish  that  its 
models  and  mechanisms  –  for  example  in  terms  of  building  a  solid  governance  for  the 
delivery of the ECOC, increasing the capacity of local cultural organizations or attracting 
projects'  ideas  from  local  residents  –  can  be  applied  not  only  within  one  individual 
Member  State  that  has  its  administrative  and  cultural  specificities  but  in  any  Member 
States, finally confirming its EU added value.  

6.  CONCLUSIONS 

The Commission concludes that the ex post evaluation of the two ECOC 2018 confirms 
what  already  emerged  from  all  previous  eleven  yearly  ECOC  evaluation  exercises,  i.e. 
that  the  ECOC  Action  is  highly  valued  by  the  hosting  cities  that  can  obtain  positive 
impacts during the development and title years. The action also remains relevant at EU 
level. 

The Commission also concludes that the programmes implemented by the two 2018 title-
holders were consistent with the objectives of the ECOC Action: 

-  They  reflected  its  European  dimension:  as  an  example,  the  programme  in 
international  collaborations 
Leeuwarden-Friesland 
covering 87 countries while Valletta 2018 had a multi-project strand dedicated to 
“Exile & Conflict” (finding a strong resonance across Europe) and developed an 
international residency programme of a scale unprecedented in the country; 

featured  overall  1,600 

-  They  involved  local  residents  and  stakeholders:  in  Leeuwarden-Friesland,  over 
60,000 local people took an active part in the delivery of the LF2018 programme 
and  there  was  an  impressive  number  of  16,159  registered  volunteers  whilst 
Valletta  developed  a  dedicated  schools  programme  that  involved  around  28,500 
children in creative projects over the period 2016-2018; 

-  They widened access to and participation in culture during 2018: audience was at 
the core of the mienskip concept in Leeuwarden-Friesland and a huge part of the 
programme (700 out of 800 projects) was developed by – and for – local people; 
in Valletta, audiences reached 400,000 people (which was less than expected) but 
the  ECOC  was  successful  in  providing  new  opportunities  for  citizens  to  be 
involved as creators, performers and audiences; 

-  They also helped strengthen the cultural capacity of the local community and/or 
cultural  and  creative  sectors  and  their  links  with  other  sectors:  for  example,  the 
LF2018  Foundation  developed  a  specific  online  toolkit  to  help  people  develop 
cultural  projects  whilst  the  ECOC  contributed  to  Valletta’s  culture-driven 
regeneration by serving as a focus for cultural infrastructure investments; 

-  The ECOC raised the international profile of Leeuwarden (albeit from a low base) 
with  7%  of  visitors  coming  from  abroad  whilst  Valletta  2018  constituted  an 
important  element  in  the  overall  Government’s  strategy  to  regenerate  and 
promote Valletta as a cultural destination world-wide; 

-  Both  ECOC-years  may  lead  to  legacies  both  physical  (in  the  form  of  new  or 
refurbished cultural and logistic infrastructures, as it happened mainly in Valletta) 
and intangible (for example through the new skills developed by the high number 

22 

 
of  local  people  who  actively  developed  cultural  projects  in  the  case  of 
Leeuwarden-Friesland). 

However, as was the case with all previous ex post evaluations of the ECOC Action, the 
current  evaluation  –  because  it  comes  too  early  after  the  implementation  of  the  ECOC 
year  in  accordance  with  Article  12  of  Decision  No  1622/2006/EC  –  cannot  assess  the 
long-term impacts  of the two ECOC 2018.  As  already mentioned in  the  Staff Working 
Document accompanying its report on the ECOC 201830, with the view to addressing this 
shortcoming,  the  Commission  intends  to  analyse  such  impacts  in  the  frame  of  the 
evaluation  exercise  foreseen  in  Article 16  of  Decision  No 2014/445/EU.  This  exercice 
will build on the results of the study published by the European Parliament in December 
2017 on "European Capitals of Culture: Success Strategies and Long-Term Effects"31. 

30  See SWD (2019) 213 final, chapter 4. 

31   http://www.europarl.europa.eu/RegData/etudes/etudes/join/2013/513985/IPOL-CULT_ET(2013)513985_EN.pdf 

23 

 
                                                           
ANNEX 1: PROCEDURAL INFORMATION 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

The evaluation was led by Directorate General for Education, Youth, Sport and Culture 
(DG EAC). It is included in the Work Programme of Creative Europe for 2018 and in the 
Agenda Planning with the reference PLAN/2018/2773. 

2.  ORGANISATION AND TIMING 

The evaluation was supported by an external and independent evaluator, under a service 
contract.  The  service  contract  was  implemented  via  a  Framework  Contract  with 
reopening  of  competition  and  in  accordance  to  the  Financial  Rules  Applicable  to  the 
General Budget of the Union32 and its Rules of Application33. 
The evaluation roadmap was adopted on 11 April 201834. 

According  to  the  roadmap,  a  Steering  Committee  including  staff  from  the  European 
Commission’s  Directorate  General  for  Education,  Youth,  Sport  and  Culture  and 
Secretariat  General  was  established  in  May  2018.  The  Steering  Committee  met  in  four 
occasions: to prepare the Terms of Reference in May, to kick off the evaluation in July 
2018, to  approve the  Inception Report in  November 2018 and to  discuss the draft final 
report in July 2019. Extensive correspondence between the Steering Committee members 
was held in between the meetings to follow-up on the evaluation.  

3.  EXCEPTION TO THE BETTER REGULATION GUIDELINES 

The  evaluation  was  exempted  from  the  obligation  to  carry  out  an  open  public 
consultation. As explained in the Roadmap, the ECOC Action is considered to be a local 
action. International participation is scattered within and outside Europe and is difficult 
to  reach.  A  wider  evaluation  exercise  will  be  carried  out  at  a  later  stage,  with  the 
objective to establish long-term impacts of the initiative as foreseen under Article 16 of 
Decision  445/2014/EU,  which  governs  the  ECOC  Action  for  the  titles  2020-2033.  The 
first  evaluation  of  this  kind  shall  be  finalised  by  31  December  2024.  A  public 
consultation will be carried out in this framework in 2023-2024 to gather widest possible 
views. 

4.  EVIDENCE, SOURCES AND QUALITY 

The 2018 evaluation of the ECOC used a series of data sets to inform its findings. The 
main ones being: 

32   http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02012R0966-20160101&from=EN 

33   http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02012R1268-20160101&from=EN 

34

file:///C:/Users/pasqusy/AppData/Local/Packages/Microsoft.MicrosoftEdge_8wekyb3d8bbwe/Te

mpState/Downloads/PART-2018-196883V1.pdf 

24 

 
 
                                                           
 
 
- 

Interviews  with  over  70  stakeholders  directly  and  indirectly  involved  in  the 
planning, operation and delivery of the two ECOC programmes; 

-  A literature review of ECOC and European level information on the two ECOC 
including  application/bid 
reports,  cultural 
programme  brochures,  web  sites  and  news  articles.  The  ECOC's  own  external 
evaluations have also been used to inform the European evaluation process; 

information,  EC  Committee 

-  Together,  the  above  evidence  base  provides  the  evaluation  with  a  valid  and 
rounded  set  of  data  to  inform  the  views  on  the  main  aspects  of  the  ECOC 
evaluation  including  efficiency,  effectiveness,  sustainability  and  relevance.  This 
view has been informed by: 
  The scale of the consultation exercise. Around 120 individuals have fed their 
views  and  opinions  into  the  evaluation  process  either  through  face  to  face 
interviews or telephone interviews; 

  The nature of the consultation exercise. The evaluators were keen to consult 
with  those  who  had  a  more  indirect  and  external  view  of  the  two  ECOC. 
These  stakeholders  including  journalists,  those  not  directly  benefitting  from 
the  ECOC  (e.g.  rejected  projects)  as  well  as  those  working  in  the  wider 
cultural policy agenda at regional and city level. This ensures the evaluation is 
not simply based on those who benefitted the most from the ECOC. 

25 

 
ANNEX 2: STAKEHOLDER CONSULTATION 

The  stakeholders  were  consulted  via  targeted  consultations  (face-to-face  interviews  or 
phone interviews).  

The consultations included the team responsible for the implementation of the ECOC, the 
political stakeholders involved in the project, the projects participating in the programme 
or  having  submitted  proposals  to  participate  that  were  rejected,  as  well  as  personalities 
attached to the cultural tissue in both cities and stakeholders at EU level. The objective of 
the  consultations  was  to  have  evidence  supporting  the  findings  and  conclusions  of  the 
evaluation. The interview lists in Leeuwarden-Friesland and Valletta, as well as the topic 
guides  for  the  various  types  of  interviews  can  be  found  in  the  Annexes  1  and  2  of  the 
report  produced  by  the  experts  assisting  the  Commission35.  The  list  of  stakeholders 
consulted was partly developed by the ECOC delivery teams but also through an internet 
search for relevant stakeholders.  

For the reasons already mentioned in Chapter 4 of the Staff Working Document and in 
Annex 1, no open public consultation was conducted in the framework of the evaluation 
exercise. The consultation was restricted to relevant stakeholders in the two cities hosting 
the title. The characteristics of the Action and the scope of the evaluation do not make it 
necessary  to  extend  the  consultation  to  a  wider  public  (as  indicated  in  the  roadmap 
published). 

Annex  3  details  the  evaluation  questions  for  which  the  stakeholders'  consultation  was 
used.  These  regard  mainly  the  questions  which  answers  are  not  based  on  factual  data. 
The  stakeholders'  consultation  was  particularly  useful  to  find  information  about  the 
impact of the ECOC in the cultural offer of the city, the participation of citizens and local 
cultural  operators,  the  building  of  capacity  for  local  cultural  operators  and  legacy 
prospects.  

35  
01aa75ed71a1/language-en 

https://op.europa.eu/en/publication-detail/-/publication/6312a17a-1b6a-11ea-8c1f-

26 

 
 
 
                                                           
 
ANNEX 3: METHODS AND ANALYTICAL MODELS 

The figure below presents the overview of the methodology. A more detailed overview 
of the methodology and sources used for each of the evaluation questions is presented in 
the subsequent tables and text. 

1. 

 Overview of methodology and tasks of the evaluation: 

Inception phase 

Task 1: Kick-off meeting  

Task 2: Initial consultations 

Task 3: Desk research  

Task 4: Inception Report  

Main Research phase 

Task 5: First visits to the ECOC cities 

Task 6: Second visits to the ECOC cities 

Task 7: Additional interviews 

Analysis and reporting phase 

Task 8: Interim Report 

Task 9: Factual check of the city reports 

Task 10: Final report 

Table 1.1  Evaluation questions: Relevance, EU added value and coherence 

Evaluation Question 

EQ1: To what extent are the objectives of the 
ECOC Action? 

What  was  the  main  motivation  behind  the  city 
bidding  to  become  a  European  Capital  of 
Culture? 

e
r
u
t
a
r
e
t
i
L

s
w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

Y 

Y 

27 

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t
y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Evaluation Question 

e
r
u
t
a
r
e
t
i
L

s
w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t
y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

the  process  of  determining 
What  was 
objectives? 
there  a  process  of 
  Was 
consultation  in  each  city  to  define  aims  and 
objectives? 

What  were  the  objectives  of  the  city  in  being 
ECOC?  What  was  the  relative  importance  of 
each objective? 

To  what  extent  were  the  objectives  consistent 
with  the  Decision  and  with  the  ECOC's  own 
application?  (special  focus  on  the  European 
dimension)  

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Have any specific objectives of the ECOC event 
been related to social impacts? 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

In  this  connection,  did  the  objectives  of  the 
ECOC  event  include  reaching  out  to  all  groups 
of 
excluded, 
disadvantaged, disabled people and minorities? 

including 

society, 

the 

EQ2:  To  what  extent  were  the  ECOC's 
associated 
cultural 
activities relevant to their own objectives? 

programmes 

and 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

To  what  extent  were  the  activities  consistent 
with the ECOC's own objectives? (special focus 
on the European dimension) 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

extent 

To  what 
specific 
themes/orientations  of  the  cultural  programme 
proved to be relevant to the objectives defined? 

have 

the 

How  was  the  European  dimension  reflected  by 
the themes put forward by the ECOC event and 
in terms of cooperation at European level? How 
did  the  Capitals  of  Culture  seek  to  make  the 
European dimension visible? To what extent did 
the two ECOC cooperate? 

28 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Table 1.2 Evaluation questions: Effectiveness 

Evaluation Question 

e
r
u
t
a
r
e
t
i
L

s
w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t
y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

EQ3:  To  what  extent  were 
objectives achieved? 

the  EU-level 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provide  typology  of  outputs,  results  and  possible 
impacts of the Action at different levels (European, 
national, regional etc.) 

To  what  extent  has 
the  ECOC  event  been 
successful in attaining the objectives of the Action 
(refer to list in the intervention logic)? 

Was the cultural programme perceived as being of 
high artistic quality? To what extent did the ECOC 
prove  successful  in  bringing  their  chosen  artistic 
themes/orientations to the fore? 

To what extent did the ECOC title contribute to an 
enhanced  cultural  offer  in  the  cities  holding  the 
title  (e.g.  in  terms  of  scope  and  scale)  with 
stronger European dimension? 

To  what  extent  did  the  ECOC  implementation 
widen access to and participation in culture in the 
two cities? What actions were taking to include the 
elderly,  young  people,  people  with  special  needs 
in the cultural activities? How accessible were the 
activities carried out? 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

did 

programmes 

the  ECOC 

help 
How 
strengthening  the  capacity  of  the  cultural  and 
creative  sectors  and  its  links  with  other  sectors? 
Which  help  was  available  to  cultural  operators  to 
extend  their  networks  and  work  transnationally 
and internationally? 

Y 

Y 

Y 

Y 

Y 

Y 

To  what  extent  did  the  Action  in  the  two  cities 
raise their international profile through culture? 

EQ4:  To  what  extent  were  the  cities's  own 
objectives achieved? 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

29 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Evaluation Question 

e
r
u
t
a
r
e
t
i
L

s
w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t
y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

What  quantitative  indicators  (number  of  visitors, 
overnight  stays,  cultural  participation  of  people, 
etc.)  of  the  social,  tourist  and  broader  economic 
impacts  of  the  event  have  been  gathered  by  the 
ECOC?  

To what extent did the ECOC achieve the outputs 
hoped  for  by  the  city  and  as  set  out  in  the 
application?  

To what extent have specific objectives related to 
social impacts been met? 

To  what  extent  were  the  objectives  related  to 
reaching out to all groups of society, including the 
excluded, disadvantaged, disabled and minorities, 
met? 

EQ5: To what extent has the Action resulted in 
unintended effects? 

Are  there  any  instances  where  the  ECOC  event 
has  exceeded  initial  expectations?  What  positive 
effects has this had?  

Where  expectations  have  not  been  met,  what 
factors  have  hindered  the  development  of  the 
Action? 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Have  any  other  unintended  effects  been 
identified? 

Y 

Y 

Y 

Y 

Y 

EQ6: To what extent can the positive effects of 
the  ECOC  Action  be  considered 
to  be 
sustainable? 

Which  of  the  activities  or  elements  of  the  ECOC 
event are likely to continue and in which form once 
the ECOC-year is over? 

Has  any  provision  been  made  to  continue  and 
follow  up  the  cultural  programme  of  the  ECOC 
event after the closure?  

How will the city continue to manage its long-term 
cultural development following the ECOC event? 

What  will  be  the  role  of  the  operational  structure 
after the end of the ECOC event and how will the 
organizational structure change? 

30 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y  

Y 

Y 

Y 

Y 

Y 

Y 

Y 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Evaluation Question 

e
r
u
t
a
r
e
t
i
L

s
w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t
y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

What  has  been  the  contribution  of  the  ECOC 
event 
improved  management  of  cultural 
development in the city? (in the medium-term) 

to 

What are the impacts of the ECOC event likely to 
be  on  the  long  term  cultural  development  of  the 
city?  
What are the impacts of the ECOC event likely to 
be  on  the  long  term  social  development  of  the 
city? 
What are the impacts of the ECOC event likely to 
be on  the  long term urban and broader economic 
development of the city? 

Table 1.3 Evaluation questions: Efficiency 

EQ7:  How  did  the  management  arrangements 
of each ECOC contribute to the achievement of 
outputs, results and impacts? 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

s
w
e
i
v
e
r

e
r
u
t
a
r
e
t
i
L

a
t
a
D

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a

h
t
i

w

s
w
e
i
v
r
e
t
n

I

Y 

s
m
a
e
t
y
r
e
v
i
l
e
d

s
n
o
i
t
a
t
l
u
s
n
o
c

l

r
e
d
o
h
e
k
a
t
S
Y 

s
t
c
e
j
o
r
p
f
o
y
e
v
r
u
S

s
w
e
i
v
r
e
t
n

i

t
c
e
j
o
r
P

s
i
s
y
l
a
n
A
Y 

How  have  the  organizational  models  of  the 
formal  governing  Board  and  operational 
structures played a role in the European Capital 
of  Culture?  What  role  have  the  Board  and 
operational  structures  played  in  the  ECOC 
event's  implementation?  At  what  stage  were 
these  structures  established?  How  did 
it 
improve  management  of  culture  in  the  city 
during the event? 

Who  chaired  the  Board  and  what  was  his/her 
experience?  What  were  the  key  success  and 
failure elements related to the work of the Board 
and  operational  structure  used  and  personnel 
involved?  

Has  an  artistic  director  been  included  into  the 
operational  structure  and  how  was  he/she 
appointed?  What  were  the  key  success  and 
failure  elements  related  to  the  work  of  the 
artistic director and personnel involved? 

Y 

Y 

Y 

Y 

Y 

Y 

What  was 
programme? 

the  process  of  designing 

the 

Y 

Y 

Y 

Y 

Y 

31 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
How were activities selected and implemented? 

How  did  the  delivery  mechanism  contribute  to 
the achievement of outputs? 

to 

been 

strategy 

To  what  extent  has  the  communication  and 
successful 
promotion 
in/contributed 
the  promotion  of  city 
image/profile,  promotion  of  the  ECOC  event, 
awareness-raising  of  the  European  dimension, 
promotion  of  all  events  and  attractions  in  the 
city? 

To  what  extent  has  the  communication  and 
promotion  strategy  including  the  use  of  social 
the 
media 
communication's 
local, 
regional,  national,  European  and  international 
levels? 

target  groups  at 

successfully 

reached 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Evaluation Question 

e
r
u
t
a
r
e
t
i
L

w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t

y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

EQ8:  To  what  extent  were  the  selection, 
co-financing 
monitoring 
procedures, 
by  Decision 
2006/1622/2006/EC efficient? 

EU 
and 
introduced 

the  Commission 

To  what  extent  have  the  mechanisms  applied 
by 
line  with  Decision 
in 
2006/1622/EC for the selection of the European 
Capitals  of  Culture  and 
the  subsequent 
implementation  and  monitoring  mechanisms 
influenced the results of the ECOC event?    

the 

To  what  extent  has 
informal  meeting 
following  the  designation  as  well  as  other 
the 
advice  offered  by 
the 
Commission 
ECOC event?  

the  panel  and  by 
the  results  of 

influenced 

L

Y 

Y 

Y 

Y 

Y 

How was the Melina Mercouri Prize used? 

EQ9:  To what  extent  did the  ECOC  manage 
to raise the necessary resources?  

Y 

Y 

Y 

Y 

Y 

Y 

32 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Evaluation Question 

e
r
u
t
a
r
e
t
i
L

w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t

y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

What was the process of securing the financial 
inputs? 

Y 

Y 

Y 

Y 

What  was  the  total  amount  of  resources  used 
for  each  ECOC  event?  What  was  the  final 
financial outturn of the year?  

Y 

Y 

Y 

What  were  the  sources  of  financing  and  the 
respective  importance  of  their  contribution  to 
the  total?  How  much  came  from  the  European 
Union Structural Funds (e.g. ERDF - European 
Regional  Development  Fund,  ESF  –  European 
Social Fund) or other sources of EU funding? 

To  what  extent  did  the  ECOC  title  trigger 
complementary sponsorship? 

What  was  the  total  expenditure  strictly  for  the 
implementation  of  the  cultural  programme  of 
the  year  (operational  expenditure)?  What  was 
the proportion of the operational expenditure in 
the total expenditure for the ECOC event?  

Y 

Y 

Y 

Y 

Y 

Y 

Y 

What  proportion  of  expenditure  was  used  for 
infrastructure 
tourism 
infrastructure, including renovation)? 

(cultural 

and 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

EQ10: To what extent were the financial and 
human  resources  secured  by  each  ECOC 
appropriate and proportionate? 

Was  the  total  size  of  the  budget  sufficient  for 
reaching  a  critical  mass  in  terms  of  impacts? 
Could  the  same  results  have  been  achieved 
with less funding? Could the same results have 
been achieved if the structure of resources and 
their respective importance was different?  

To  what  extent  have  the  human  resources 
deployed for preparation and implementation of 
the  ECOC  event  been  commensurate  with  its 
intended outputs and outcomes?  

Y 

Y 

Y 

Y 

Y 

Y 

Y 

33 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Evaluation Question 

e
r
u
t
a
r
e
t
i
L

w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t

y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

s
i
s
y
l
a
n
A

be 

As a result, could the total budget for the ECOC 
event 
and 
proportional to what the each ECOC set out to 
achieve?  

appropriate 

considered 

Y 

Y 

Table 1.4 Evaluation Questions: Coherence 

Evaluation Question 

e
r
u
t
a
r
e
t
i
L

w
e
i
v
e
r

e
v
i
t
a
t
i
t
n
a
u
Q

s
i
s
y
l
a
n
a
a
t
a
D

h
t
i

w
s
w
e
i
v
r
e
t
n

I

s
m
a
e
t
y
r
e
v
i
l
e
d

l

r
e
d
o
h
e
k
a
t
S

s
n
o
i
t
a
t
l
u
s
n
o
c

f
o
y
e
v
r
u
S

s
t
c
e
j
o
r
p

t
c
e
j
o
r
P

s
w
e
i
v
r
e
t
n

i

EQ11:  To  what  extent  were 
complementary to other EU initiatives? 

the  ECOC 

Y 

Y 

Y 

Y 

s
i
s
y
l
a
n
A

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

As far as the conclusions made for the two cities 
allows it, to what extent has the Action proved to 
be  complementary  to  other  EU  initiatives  in  the 
field of culture? 

To what extent has each  ECOC been reinforced 
by and added  impetus to investments by the EU 
Structural Funds? 

To  what  extent  have 
complemented 
European Year of Cultural Heritage? 

other  EU 

the 

two  ECOC 
e.g. 

initiatives, 

EQ12:  What  is  the  EU  added  value  and  the 
visibility of the ECOC Action? 

As far as the conclusions made for the two cities 
allow,  what  is  the  added  value  of  the  European 
Capital  of  Culture  being  an  EU 
initiative, 
compared to what could be achieved if the Action 
was a purely national or local action? 

Was the fact that this is a European Union action 
sufficiently communicated by the cities?  

To  what  extent  were  the  general  public  and  the 
cultural operators aware of this fact? 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Note: Project surveys were conducted directly by the two ECOC cities. 

34 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2.  Details of the relevant evaluation tasks and methodologies:  

Inception phase 

The  aim  of  the  inception  phase  was  to  finalise  the  evaluation  framework  and  research 
tools, address the project steering group's comments regarding the proposed method and 
collect background information on the two 2018 host cities. 

Initial consultations   

The  task  consisted  in  talking  to  each  of  the  two  ECOC,  getting  evidence  about  the 
efficiency  and  effectiveness  of  processes  at  EU  level,  obtaining  qualitative  information 
on key issues for each ECOC to inform later research, identifying additional sources of 
data  and  additional  interviewees  at  EU,  local  and  national  levels,  as  well  as  informing 
proposals on monitoring and evaluation arrangements. Discussion at the kick off meeting 
identified  a  need  for  strong  triangulation  and  to  gain  opinions  from  a  broader  set  of 
stakeholders. 

Desk research 

This involved collection and review of literature, primary data from the delivery agency 
and  available  secondary  data,  as  well  as  web  and  social  media  statistics  in  order  to 
develop a descriptive picture of each ECOC and serve as a source of evidence for later 
analysis.  The  list  of  the  literature  that  was  collected  at  the  European  level  which  was 
reviewed and assessed for usefulness for the sake of the evaluation can be found in the 
Annex  3  of  the  Report36.  Much  of  the  European  level  literature  is  helpful  to  either 
understand the policy drivers at EU level which support the two ECOC 2018 or provide 
background  on  the  bidding  and  application  stages  for  both  2018  cities  (i.e.  little  of  it 
provides information useful for the content, delivery and impact of the two programmes). 
More academic literature at the EU level linked to the ECOC was also searched  for. To 
complement  the  above  desk  research  task  the  contractor  assessed  the  extent  to  which 
each  ECOC  had  used  big  data  as  well  as  analysed  web  and  social  media  to  increase 
visibility  and  interest  in  the  ECOC  among  country  residents  and  internationally.  There 
was however no evidence of big data having been used by the Foundation or any of the 
other stakeholders in Leeuwarden-Friesland and Valletta 

First visits to the cities  

Initial face to face visits were made, to  Valletta and to Leeuwarden in late 2018. These 
interviews sought to gather an in-depth understanding of the effect that ECOC had for the 
individuals  involved, their organizations  and the local  culture sector  as a  whole.  It  also 
allowed to ensure that findings are based on consensus across a range of target groups as 
well as to identify key differences in the experience of different stakeholders. In the case 
of  projects,  the  criteria  for  inclusion  focused  on  whether  it  represents  good  practice  in 
relation to at least one of the key dimensions of the evaluation, including (but not limited 
to) increased European cooperation, the  effective targeting of key  groups of citizens or 
neighbourhoods, audience development strategies, or legacy effects. 

36   See previous footnote.  

35 

 
                                                           
Second visits to the cities 

With a clear view emerging from desk research as to the overall relevance, efficiency and 
effectiveness  of  each  ECOC,  the  contractor  undertook  a  second  three  day  visit  to  each 
city  in  spring  2019.  The  visit  focused  on  conducting  in-depth  interviews  with  the 
managing teams, their key stakeholders and cultural operators identified during the initial 
visits.  

Further interviews were conducted: 

  with national and local  media representatives in order to  assess media strategies 

 

and impact on communication efforts; 
representatives  of  local  commerce,  including  businesses  in  the  tourism, 
hospitality  and  transport  sectors  as  well  as  key  commercial  sponsors  of  ECOC; 
and  

  other national  ministries  or regional authorities to  explore regional  development 

issues. 

Building on the interviews undertaken in the first visits, the interviews with the delivery 
teams  served  four  purposes:  i)  gathering  further  factual  data  and  information  about  the 
cultural  programme  and  its  achievements;  ii)  identifying  the  "story"  of  the  ECOC 
throughout  its  lifecycle,  i.e.  conception,  application,  development,  delivery,  legacy;  iii) 
gaining a critical (albeit "insider") perspective on the relevance, efficiency, effectiveness 
and sustainability of the ECOC; iv) triangulating emerging findings, for example, those 
emerging from the desk research. 

Interviews  with  other  key  stakeholders  in  the  city  (e.g.  municipality,  chamber  of 
commerce, commercial sponsors, media representatives and cultural operators) were also 
essential to identifying the ‘external’ view of the ECOC by those who were involved in 
the urban or cultural  agenda but  were not  necessarily involved in  the direct  delivery of 
the ECOC. As with the previous evaluations, such individuals offered an alternative and 
broader  perspective  on  the  ECOC.  Indeed,  the  interviews  allowed  the  evaluator  to 
explore particular issues in more depth, for example, relating to the effectiveness of the 
governance structure, or the strength of artistic direction. 

Analysis and reporting phase 

Once  all  the  information  gathered,  the  experts  fulfilled  an  analysis  of  the  available 
information,  drawing  conclusions  and  triangulating  data  to  ensure  consistency  and 
accuracy. The limitations of the data gathered have been explained in  the report and in 
the  Staff  Working  Document.  After  the  first  drafting  exercise,  the  results  were  shared 
with  the  two  ECOC  for  a  factual  check.  Both  the  Valletta  and  Leeuwarden-Friesland 
ECOC  were  asked  to  undertake  a  check  on  the  completeness  of  data  being  used  to 
address  each  evaluation  topic.  Their  comments,  where  relevant,  were  included  in  the 
Final Report submitted by the contractor supporting the evaluation. 

36