EUROPEAN COMMISSION Brussels, 18.8.2020 SWD(2020) 165 final COMMISSION STAFF WORKING DOCUMENT Ex-post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta) Accompanying the document Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions

Ex Post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta)





{COM(2020) 386 final}



Table of Contents 1.Introduction 2.Background to the intervention 3.Implementation / state of Play 4.Method 5.Analysis and answers to the evaluation questions 6.Conclusions Annex 1: Procedural information 1.Lead DG, Decide Planning/CWP references 2.Organisation and timing 3.Exception to the better regulation guidelines 4.Evidence, sources and quality Annex 2: Stakeholder consultation Annex 3: Methods and analytical models



COMMISSION STAFF WORKING DOCUMENT Accompanying the document

Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions Ex Post evaluation of the 2018 European Capitals of Culture (Leeuwarden and Valletta) 1.Introduction This document describes the methodology and findings of the ex-post evaluation of the European Capital of Culture (ECOC) Action for 2018. Article 12 of Decision No 1622/2006/EC (here after, the “Decision”) requires that the Commission ensures the external and independent evaluation of the results of the ECOC from the previous year. The aim of the evaluation is to better understand how the two European Capitals of Culture of the previous year contributed to the objectives of the Action, whether they achieved their objectives and whether the implementation proceeded in line with their original application. It is also to contribute to reinforcing the existing evidence-base on the ability of the Action to produce cultural, social and economic impact. Finally, the objective is to draw lessons, conclusions and recommendations that may be useful for Union institutions, future ECOC or cities wishing to bid for the ECOC title in the future. The Commission started these yearly evaluations with the 2007 title-holders and the evaluation of the ECOC 2018 is the latest to date in a series of 12 similar yearly exercises. This staff-working document is the fourth covering the ECOC Action in as many years, while the Action in itself has not changed in the period considered, with only the pair of cities hosting the Action being different each year. This staff-working document summarizes the findings of the external evaluation of the implementation of the ECOC 2018, including the selection and monitoring procedures as well as the operational delivery by the two hosting cities, namely Valletta in Malta and Leeuwarden in the Netherlands . The evaluation examined how the two cities developed their respective applications and cultural programmes, how they delivered the planned initiatives throughout the year and any cultural, general and longer-term impact generated by the Action. The evaluation constitutes a valuable opportunity to reconsider critically the past year with the intention of collecting further insight and lessons based on the experiences of the host cities. It is also important to underline that the aim of the yearly evaluation is not to lead to any change in the regulation governing the ECOC Action. Indeed, those rules changed in 2014 following an ad hoc interim evaluation of the selection and monitoring procedures of European Capitals of Culture , but the new rules only apply for cities designated as ECOC for the years from 2020 to 2033 . It would therefore be premature to use the outcomes of the evaluation of the two ECOC 2018, governed by previous Decision No 1622/2006/EC, to draw conclusions related to Decision No 445/2014/EU, which introduced changes in particular regarding the selection procedure, monitoring arrangements or the payment of the Melina Mercouri Prize. 2.Background to the intervention 2.1.The European Capital of Culture Action The initial scheme of "the European City of Culture" started at an intergovernmental level in 1985 upon an idea of the then Greek Minister of Culture, Melina Mercouri. The scheme recognized Europe as a centre for artistic development, underpinned by an exceptional cultural richness and diversity, with cities playing a vital role in the formation and spread of cultural expressions. In 1999, Decision 1419/1999/EC of the European Parliament and the Council gave the scheme the status of a Community Action under the name of "European Capital of Culture" and set up a more predictable, consistent and transparent system for the designation of hosting cities. Its legal foundation was Article 151 of the Treaty (now Article 167), which calls on the Union to “contribute to the flowering of the cultures of the Member States, while respecting their national and regional diversity and at the same time bringing the common cultural heritage to the fore”. The Decision introduced new selection procedures and evaluation criteria while Member States were ranked in a chronological order of entitlement to host the event each year. In 2005, the European Parliament and the Council adopted Decision 649/2005/EC to integrate the ten countries that joined the European Union in 2004. In 2006, a new Decision – Decision 1622/2006/EC – entered into force. This Decision kept the principle of a chronological order of Member States but further refined the objectives of the Action and introduced new selection and monitoring arrangements. Decision 1622/2006/EC is the legal basis governing the European Capital of Culture event for the titles 2007 to 2019. As mentioned under section 1, in April 2014, the European Parliament and the Council adopted a new Decision, but cities designated as ECOC for the years up to 2019 continue being regulated by Decision No 1622/2006/EC. The 2014 Decision was, in turn, amended in September 2017 to open the ECOC Action to European Free Trade Association countries that are parties to the Agreement on the European Economic Area . 2.2.Objectives of the ECOC Action The ECOC Action aims to highlight the richness and diversity of European cultures and the features they share, thereby promoting greater mutual understanding among European citizens, as well as to foster the contribution of culture to the long-term development of the cities. ECOC shall strive to foster cooperation between cultural operators, artists and cities in Europe, foster the participation in cultural activities of the citizens living in the city and surroundings while raising the interest of citizens from abroad, to be sustainable and to be an integral part of the long-term cultural and social development of the city.

The hierarchy of objectives presented in the table below is based on the objectives as stated in Decision No 1622/2006/EC, but it has been updated to reflect the content of the new legal basis for ECOC post-2019. The general and strategic objectives are taken directly from Article 2 of Decision No 445/2014/EU, with the operational objectives flowing logically from these. They are also informed by the selection criteria detailed in Article 5 of the 2014 Decision. Table on ECOC hierarchy of objectives General objective Safeguard and promote the diversity of cultures in Europe, highlight the common features they share, and foster the contribution of culture to the long-term development of cities Specific objectives (SO) SO1: Enhance the range, diversity and European dimension of the cultural offer in cities, including through transnational co-operation SO2: Widen access to and participation in culture SO3: Strengthen the capacity of the cultural and creative sector and its links with other sectors SO4: Raise the international profile of cities through culture Operational objectives Stimulate a diverse range of cultural activities of high artistic quality Implement cultural activities promoting cultural diversity, dialogue and mutual understanding Implement cultural activities highlighting (shared) European cultures and themes Involve European artists, promote cooperation with different countries and transnational partnerships Create new and sustainable opportunities for a wide range of citizens to attend or participate in cultural events Involve local citizens, artists and cultural organizations in development and implementation Provide opportunities for volunteering and foster links with schools and other education providers Improve cultural infrastructure Develop the skills, capacity or governance of the cultural sector Stimulate partnership and co-operation with other sectors Combine traditional art forms with new types of cultural expression Attract the interest of a broad European and international public 3.Implementation / state of Play 3.1.The selection and monitoring of the European Capitals of Culture 2018 In accordance with Decision No 1622/2006/EC, Malta and the Netherlands were the two Member States entitled to host the ECOC in 2018. Under this Decision, host countries are responsible for the procedure leading to the selection of one of their cities as "European Capital of Culture". This is done through an open competition within the Member State concerned. Against this background, the relevant managing authorities of Malta and the Netherlands, respectively the Maltese Ministry of Tourism, Culture and the Environment and the Dutch Ministry of Education, Culture and Science, run parallel competitions, which both started with the publication of a call to trigger applications from interested cities. The selection is then in two phases: a pre-selection round (candidate cities are reduced to a short-list) followed by a selection round (one city is recommended for the title). A panel of thirteen members – six of whom nominated by the Member State concerned and the other seven by European Union institutions and bodies (European Parliament, Council, Commission and Committee of the Regions) – examines the bids from candidate cities based on the objectives and criteria laid down in the Decision. Malta decided to start its selection procedure one year ahead of the normal time schedule because it knew that major infrastructure projects – which need long planning periods – were necessary to help the country host an event of the scale and scope of a European Capital of Culture. It therefore published its call for submission of applications in December 2010, seven years before the start of the ECOC-year while the rule is to start the competition “no later than six years before the event is due to begin”. The main stakeholders at national and local levels had taken an early decision that an application would be submitted on behalf of Valletta but involving the entire territory of the Maltese islands. Valletta’s application was consequently the only one submitted by the call’s deadline of 17 October 2011. Following the pre-selection meeting in January 2012, the city was short-listed and in November 2012, the panel recommended that Valletta be awarded the ECOC 2018 title in Malta . The Netherlands published their call in December 2011, in line with the normal time line. Five cities entered the competition by the call’s deadline of 31 October 2012: Eindhoven, Leeuwarden, Maastricht, The Hague and Utrecht. At the pre-selection meeting in November 2012, three of them (Eindhoven, Leeuwarden and Maastricht) were short-listed. In September 2013, the panel recommended that the ECOC 2018 title in the Netherlands be awarded to Leeuwarden. The Council of the European Union formally designated Valletta and Leeuwarden as ECOC 2018, respectively in May 2013 and in May 2014 . In line with Decision No 1622/2006/EC, once designated as ECOC and until the title-year, the two cities had to adhere to a monitoring procedure directly managed by the Commission. All along this phase, the progress in the cities' preparations was monitored and guided by a panel composed of the seven independent experts appointed by the European Union institutions and bodies, which also checks compliance with the programme and commitments on the basis of which the two cities had been selected. As part of this process, Valletta and Leeuwarden submitted progress reports and attended two formal monitoring meetings with the panel that the Commission convened in November 2015 and April 2017. On top of that, the Commission organized informal post-designation meetings (that do not result in public reports, as they do not have a formal status) between the panel and Valletta in October 2013 and in November 2014 and between the panel and Leeuwarden in November 2014. An in situ visit (whereby a delegation of the panel and staff members of the Commission visit a hosting city to get a better insight of the development of the project) took place in Valletta in January 2016. No similar visit was deemed necessary in the case of Leeuwarden as preparation progressed more smoothly. During this monitoring phase, the panel made a number of recommendations to the two cities, in particular to ensure that they adequately involve all relevant groups of citizens, develop an ambitious European dimension and reflect on a proper legacy of their ECOC-year. As was the case for all previous ECOC, during the development period, the two cities introduced modifications into the programme described in their original applications, either in response to a changing environment or to the recommendations made by the panel. However, the changes made by Valletta were of a much bigger scale than the usual practise, as the initial concept of the bid was dropped following a change of leadership after the country’s 2013 general election. The panel took note of these changes and underlined the importance to preserve the artistic coherence of the project in line with the application’s original objectives. At the last meeting, while praising Valletta for the good preparatory work in terms of monitoring, evaluation and involvement of the various communities in the Maltese islands, the panel highlighted the need to simplify the overall narrative, continue efforts to gather sponsors, bring forward legacy discussions and business plans for flagship venues and clarify targets for international visibility. As far as Leeuwarden is concerned, at the end of the monitoring process, the panel praised the excellent work done in bringing the project forward, underlining in particular its strong European dimension, the high degree of involvement of local citizens and the focus on issues of global and European reach closely linked to current challenges. It however recommended that Leeuwarden increase the project’s international visibility, including through cooperation with ad hoc national authorities, and involve all relevant stakeholders, from institutions to civil society at regional and local levels, in governance discussions in order to develop a sustainable legacy strategy.

The monitoring process culminated with the panel making a positive recommendation to the Commission on awarding the €1.5m prize in honour of Melina Mercouri to both cities after the last monitoring meeting in April 2017. Based on this positive recommendation, in the autumn 2017, the Commission financed the prize to the two ECOC 2018 from the budget of the EU Creative Europe programme . The sub-sections below describe the main features of the ECOC programmes of Valletta and Leeuwarden. The final report of the contract supporting the evaluation contains further information, including the way the programmes had a European dimension and involved citizens (as these are the main two criteria of the ECOC Action) as well as on the development of the applications and the governance and funding structures. 3.2.Valletta 2018 Valletta is the capital city of Malta and part of a wider metropolitan area of nearly 400,000 people, representing 90% of the country’s population. The area is heavily dependent on leisure tourism, whilst other key industries include medical tourism, electronics, textiles and film production. The country’s history has given Valletta a rich cultural and architectural heritage, which enabled the city to gain UNESCO World Heritage status in 1980. Valletta also has a long tradition in the fine arts. This led to the creation of the National Museum of Fine Arts, which operated from 1974 to 2016. The museum has now been replaced by a new institution, MUŻA (an acronym for “MUŻew Nazzjonali tal-Arti” in Maltese), which had a partial opening towards the end the title-year of Valletta 2018. Other notable aspects of Valletta’s cultural life include the Maltese Carnival as well as the Valletta International Baroque Festival. The initial overall slogan of Valletta 2018 was “Imagine 18”, which had emerged from the various consultation activities that had taken place during the application phase. The intention was to trigger positive responses supporting the ECOC project, inspire Maltese society and appeal to overseas audiences and partners. The programme was articulated around four main themes (Generations, Routes, Cities and Islands) and pursued four main objectives: Making careers of culture (supporting the development of artists and creatives and growing the critical mass of audiences); Growing internationally (engaging across and beyond Europe’s borders); Establishing Valletta as a creative city (targeting the city’s social, economic and cultural regeneration) and Nurturing sustainable relationships with the environment (based on new sustainable connections between citizens, architecture and landscape). In this way, Valletta 2018 was well connected with a strategic development plan for Valletta and for Malta. It also had the unanimous support of all 68 local Councils in the country and cross-party political support in Parliament. Early in the development phase, the initial concept of “Imagine 2018” was however considered as being of limited relevance or usefulness and it was eventually dropped in favour of the strapline: “Valletta 2018: An island-wide fiesta”, reflecting the intention for the ECOC to better represent the whole of Malta and facilitate the participation of citizens across the national territory. Indeed, this theme – and corresponding thematic lines (Island Stories, Future Baroque and Voyages) – was chosen for its reference to Malta’s local traditions and fiestas bringing whole communities together. At the time of the final monitoring meeting, Valletta 2018 reported, however, that 50 out of the 64 projects included in the initial application were still underway. The responsibility for the organization and implementation of the ECOC project was entrusted to a new body, the Valletta 2018 Foundation, which had been created to develop the application. The Foundation was led by a Board of Governors consisting of ten members, including the Mayor of the City of Valletta, the Rector of the University of Malta, the President of the Malta Chamber of Commerce, Enterprise and Industry and other stakeholders. The Foundation was also supported by a wider partnership that included all 68 Local Councils (which each signed a charter), the Association of Local Council and Arts Council Malta. It is worth noting that during the development phase, the Foundation went through some considerable changes in personnel, including at senior staff level in 2013, and again in 2017. Moreover, the governance of the ECOC became the subject of much political debate and disagreement. In particular, it attracted a considerable degree of national and international criticism following comments posted on social media by the Chairman, which appeared to mock the murdered journalist, Daphne Caruana Galizia. As far as budget is concerned, the figure indicated in the application amounted to €49.57m for the years 2013-20. The eventual operational budget was nevertheless much lower at €26.5m, €17.31m of which dedicated to the cultural programme itself. Most of the funding (€24.22m) came from the national Government. However, a positive point is that it seems a considerable proportion of this contribution consisted of new funds, while in the application a majority of the national Government’s contribution was supposed to come from reallocated funds as opposed to new funds (€21.23m out of a total of €39.67m). The budget for the cultural programme was also complemented by significant Government investment in cultural infrastructure. Finally, sponsorship raised €60k, which was much less than originally intended (€2m). The EU funding provided in the form of the Melina Mercouri Prize was added to the general budget for operating expenditure. 3.3.Leeuwarden 2018 Located in the northwest of the Netherlands, Leeuwarden is a city of 100,000 inhabitants in the region of Friesland, which has a population of 646,000 and a total of eleven historic cities connected together by water. Leeuwarden is also one of the oldest cities in the north of the Netherlands with a rich history dating back to the Roman age. Friesland has two official languages (Dutch and the local Frisian) and is the only bilingual region in the Netherlands with Frisian being taught in many of the region’s schools. The inhabitants of Friesland are also recognised as having a strong cultural identity and sense of local pride. The city and the region are relatively well served as far as cultural infrastructures (such as the impressive Fries Museum that opened in 2013) and activities (including big festivals) are concerned, but their cultural sector is often seen as being locally-focussed and inward-looking in terms of cultural content, target audience and ambition. Leeuwarden associated the Friesland region in its bid, a key element of which was “Iepen (open) mienskip”. Mienskip is a Frisian word and tradition, associated with core values such as mutual respect, participation, grass-roots development, equality and civic responsibility. The adjective “Iepen” adds an important second element of “openess” and expresses the need for the city and the region to welcome external input and become more open-minded and outward-looking in including those not belonging originally to the “Frisian community”. The open mienskip approach was to dominate the development work of Leeuwarden-Friesland 2018, as well as the content of its cultural programme, with a key overarching principle being around using culture to help deal with society’s issues. Tackling social issues was indeed a key part of the bid-book and eventual cultural programme of the ECOC-year, particularly in the fields of social inequality (such as child poverty), ecology (such as the threat to biodiversity or water management), diversity (notably, minority manguages and the integration of migrants) as well as the relationship between the city and the countryside, which are challenges that are all common to many other cities and regions in Europe. Against this background, the cultural programme was organised around the following five thematic lines: Nature and Culture (using culture as a way to draw attention to nature); City and Countryside (exploring themes such as brain drain from countryside to cities or reflecting on how to create creative ecologies); Community and Diversity (promoting connections between people from different cultural backgrounds); Lab Lws (using culture as a vehicle for innovation) and Royal Frisians (highlighting the rich culture of the area). More specifically, three story lines were explored across these above overarching themes: Dare to Dream (bringing the city and region to the European stage), Dare to be Different (opening the Frisian community to different cultures and Europe) and Dare to Act (protecting the unique biodiversity of the Frisian region). As with other ECOC, a dedicated delivery entity, the LF2018 Foundation, was set up as an independent body to specifically develop and deliver the programme. It consisted of around 60 staff members linked to a range of roles including marketing and communication, internationalisation, production as well as finance. The Foundation’s Supervisory Board was independent of the region and city authorities and consisted of five members selected from different perspectives such as knowledge areas (artistic, legal, governance). The representatives of the region and city were connected through a regular formal meeting with the CEO and managing director of the LF2018. The original budget foreseen at the bid-book stage for the delivery of the ECOC was estimated at €74m, with the municipality of Leeuwarden, the other municipalities in the region and the province of Friesland contributing respectively €5.95m, €11m and €20m. The Dutch Ministry of Education, Culture and Science and the Dutch Ministry of Interior Affairs pledged to contribute €7.5m. Expected income from private sponsors was estimated at a comparatively high level, at €16m or 22% of the total budget. By the end of 2018, the actual budget had increased to €104.6m (of which €84.4m for the cultural programme), meaning spend on the ECOC increased by €30.3m compared to the bid-book. This increase in budget was mainly the result of an increased income from the private sector (€50.8m instead of €22m) as ticket sales peaked at €20m instead of the €4 million planned. 4.Method In order for results to be comparable with previous evaluations, the methodology for this evaluation closely followed the approach adopted in previous assessments of the Action . The focus has been on research at city level and, in particular, the gathering of data and stakeholders' views from both Valletta and Leeuwarden. The main evaluation sources can be identified as follows: -EU level literature: this included higher level EU policy and legislative briefings, papers, decisions and other documents relating to ECOC. This mainly focussed on reports of the panels for selection and monitoring and the original bidding guidance to understand how the two ECOC established themselves in the early days. Academic research was also consulted regarding the ECOC Action and the role of culture in the development of cities; -ECOC level literature from Valletta and Leeuwarden-Friesland: this included original bids and applications, internal reports linked to the application, monitoring or evaluation processes, as well as other pieces of literature analysing the cultural programme itself; -Quantitative data: where available, evidence linked to each ECOC was collected in relation to budgets and expenditures, projects' numbers and types, participation and audience figures as well as other pieces of quantitative data relevant to the assessment of the work and benefits of the ECOC in each city;

-Interviews with managing teams: those responsible for the day-to-day design and delivery of the ECOC were interviewed in each city during city visits in late 2018 (i.e. during the host year) and in spring 2019 (13 interviews of the Leeuwarden team and 10 interviews for the Valletta team). Almost all of the key individuals within the delivery agencies were interviewed including those linked to strategic development, marketing and communication, project implementation and financial management; -Interviews with other key stakeholders : mainly face-to-face interviews were undertaken with stakeholders both directly and indirectly involved in either the planning or delivery of the ECOC along with those more widely linked to the cultural, social, economic or political agenda of the host cities. Stakeholders included those working in cultural organisations, city/regional/national administrations, tourism and visitor agencies, media organisations as well as voluntary and community organisations. Interviews were also conducted with managers of individual projects and activities supported through the ECOC Action that made up the cultural programme of each city. A number of international artists and cultural operators were interviewed, which provided an external perspective on the ECOC; -Interviews with EU-level stakeholders: two former members of the selection and monitoring panel gave feedback while EU-level cultural bodies were invited to contribute but only one responded; -The evaluation does not include a wider public consultation. As explained in the roadmap, the Action is considered to be local while international participation is scattered within and outside Europe and is difficult to reach. On the other hand, the opinions would be based on attendance to specific events and would not give useful insights for the evaluation of the ECOC Action as a whole. The final report of the contract supporting the evaluation provides a detailed understanding of the 2018 ECOC Action and within this an assessment of the work and progress of Valletta and Leeuwarden. There are, however, issues to consider when assessing the strengths of the evidence base used for this study: -Although both cities have undertaken some form of evaluation work themselves, part of the results of these studies were not yet available at the time of the evaluation. The European evaluation of the ECOC Action has used as much of this secondary information as possible, but could not benefit from its final results, especially for what concerns quantitative data; -As already mentioned in previous evaluations, an ideal way of conducting this evaluation would entail a before ('baseline') study and an after-picture ('ex-post') study, instead of carrying out the latter alone; the impossibility of comparing the two studies affects the accuracy of the evaluation. Moreover, since the evaluation was undertaken during and shortly after the end of the ECOC time framework some of the effects of the programme had not manifested themselves entirely. Many stakeholders involved in the evaluation commented that the real impact of the ECOC on the city and its residents would take time to filter through. In this respect, the Commission once again highlights that budget and timing only allow an ex-post evaluation to take place and therefore only an after picture has been studied; -An ulterior consequence of the modest yearly budget allocated to the evaluation is the fact that the primary evidence data gathering tends to be more of qualitative than quantitative nature; while qualitative data still holds a great importance in the evaluation, the lack of diversity of data sources translates into a lesser dependability, for instance, in the process of proving the objective outcomes and impacts of ECOC on widening participation in culture; -Detailed modelling, economic impact assessments or large-scale surveys were outside the scope of this study. This evidence used outside of the interviews is dependent on the local evaluation and other research commissioned by the ECOC cities. Both 2018 ECOC cities did not commission large and ambitious evaluations providing quantitative data or researching the economic impact of the programme or levels of cultural awareness before, during and after the ECOC year. 5.Analysis and answers to the evaluation questions This part of the report presents a diagnosis of the implementation of the ECOC Action in 2018. It draws together the results relating to the two ECOC 2018 and to a lesser extent the findings from previous ECOC evaluations. All figures appearing in the following sub-sections come directly from the report of the external ex-post evaluation of the two ECOC 2018. The reader will find in the full text of the document many more examples illustrating the conclusions presented below. Very often, examples and figures can illustrate different points made under "relevance", "efficiency", "effectiveness", "sustainability", "EU added value" and "coherence". They are therefore sometimes repeated in various sub-sections. However, for the ease of the reading, repetitions have been limited to the minimum and the reader is invited to correlate some conclusions presented below with examples or figures that may have been given in previous paragraphs or may be given in following ones. 5.1.Relevance Findings from the ECOC 2018 evaluation show that the two title-holders developed and implemented cultural programmes that were consistent with the EU Treaty, in particular Article 167 , as well as with Decision No 1622/2006/EC. Indeed, the ECOC Action was a good vehicle for the two hosting cities in 2018 to promote cultural diversity and highlight common elements, contributing in this way to the "flowering" of the local, national and European cultural scene. As is shown in following sub-sections through many concrete examples, the two cities organized thousands of events on top of their usual cultural offer. These events covered all types of cultural and artistic disciplines, targeted many groups of audiences and presented various aspects of the diversity and common features of cultures in Europe, notably through co-operations with partners from other European countries and beyond. It is worth noting that despite considerable changes in the artistic themes, governance and other aspects of Valletta 2018 during the development phase (as mentioned before), the objectives of the revised programme remained broadly consistent with the spirit of three of the four original objectives set in the original application (with less focus being given to the fourth relating to the environment) and thus with the objectives set at EU level. This is illustrated in the sections below. The European dimension in Valletta At the time of the selection, the panel did not consider that Valletta 2018 had a particularly strong European dimension. During the development phase, the panel thus made recommendations to help the ECOC delivery Foundation improve it, in particular through the themes put forward in individual projects, intensified co-operation with European artists and cultural operators and increased efforts to make Maltese citizens more aware of Europe and European citizens more aware of Malta. In reaction to such recommendations, the Foundation took specific measures, in particular the appointment of an International Artistic Advisor, who proved very instrumental in restructuring the programme around a stronger international dimension and in connecting the Foundation and Malta’s cultural stakeholders to international artists. The view of the evaluator is that by the end of the ECOC-year, the cultural programme of Valletta 2018 featured more activities promoting the cultural diversity of Europe and highlighting the common aspects of European cultures than Malta’s cultural offering in previous years. Explicit objectives were set relating to the European (and international) dimension, such as integrating Malta’s rapidly-changing and diverse society into the programme, as well as engaging communities across and beyond Europe’s borders. A good illustration is the development of projects exploring the theme of European migration as part of a multi-project strand (added during the preparatory phase as a response to the panel’s request), “Exile & Conflict”, which featured several projects, or the production of new opera entirely in Maltese, “Aħna Refuġjati”, addressing the very topical issue of large-scale migration across the Mediterranean. There were also more collaborations with European artists and operators from different countries. New international partnerships were established and the cultural programme featured more performances by international artists and exhibitions of international works and residencies by European artists. According to research carried out locally, over 200 international artists were to be part of the programme, from countries such as Egypt, Tunisia, Syria, Jordan and Lebanon (to reflect the Mediterranean dimension of the ECOC), as well as Europe and the rest of the world. As an example, the “Meeting Point” strand (in charge of connecting projects, people, disciplines and genres) arranged eight residency programmes across Malta, which enabled 50 European artists to complete residencies in the country, as well as three outgoing residencies and three international workshops. However, several international artists responsible for key projects reported that their overall experience of Valletta 2018 had been somewhat negative due to challenges in governance (see above). Similarly, the political controversies associated with Valletta 2018 had the effect of attracting negative international attention and damping the enthusiasm of some international partners.

Finally, the tourism data (particularly the increased cultural motivation to visit Valletta) suggests some success in reaching a European public (see below under “effectiveness in reaching out European and international audiences”). The “City and Citizens” dimension in Valletta In practice, the main effort was put into connecting the ECOC with citizens and audiences across Malta. This was a key element of Valletta 2018’s application and this remained a strong focus of the Foundation’s work all along the development phase of the project. The initial idea was that the ECOC project would encompass the whole territory of the Maltese islands, and to do that, it was decided to recruit a team of 6 regional co-ordinators, with the responsibility to undertake community outreach for different territories. In the evaluator’s view, this proved an interesting element in the Foundation’s governance. On top of that, one of the sub-sections of the Valletta 2018 programme featured 13 projects specifically focussed on involving children and young people from across Malta, as these were supposed to be primary targets of the ECOC. In particular, work was done directly with schools, with a dedicated programme that involved around 28,500 children in creative projects over the period 2016-2018. Finally, there was some activity, albeit relatively limited, to connect with and involve the Maltese diaspora internationally, such as a site-specific installation and video work, bringing the observations and recollections of the Maltese diaspora to an audience in Malta. The European dimension in Leeuwarden According to the evaluator, Leeuwarden-Friesland 2018 took the European dimension very seriously. An extremely important point in this regard is that while other ECOC tend to ensure that staff in general consider the European dimension with no overall lead in charge of pro-actively pushing this agenda forward, a key member of the LF2018 team was specifically responsible for promoting and embedding the European dimension at both programme and project levels. The responsibility of this person was to check, monitor and encourage project owners, the ECOC Board and fellow management team members to promote a European aspect to their daily work and thinking. As a result of this strong commitment, there were overall 1,600 international collaborations covering 87 countries, which was five times greater than originally planned. An example is the “11 Fountains” project – 11 artists from across the world were commissioned to design and build 11 new fountains with local residents in each of the 11 Frisian cities to promote the importance of water in the region. Another telling illustration is the “Potatoes go wild” project, which consisted in a series of cultural activities that involved artists from Malta, Germany, Ireland and Serbia using the common theme of the potato to share European stories. The ECOC programme also addressed a number of key “European themes” that linked Leeuwarden-Friesland with other regions across the EU Member States facing similar challenges and issues (e.g. minority languages, migration, urban/rural pressures as well as a range of environment and ecological issues). A good example is the “Lost in the greenhouse” project, which focused on migration in Europe and included performances produced with local people and migrants living in the area, especially Polish workers. It covered a range of European issues including disparities in wage levels across the EU, skilled labour moving to Western Europe, integration issues with newly arrived migrants as well as the economic consequences and benefits of migration in different countries. Another strong aspect of Leeuwarden-Friesland’s approach to the European dimension was the existence of funding to make it easier for projects to find partners from across Europe. Some €265k from the open programme (i.e. the part of the cultural programme that dealt with the bottom-up mienskip approach) was ring-fenced to help smaller organisations find European partners, for example, by covering the costs of visits. The ECOC in Leeuwarden-Friesland also had a number of processes in place to check, report and monitor the extent to which the programme and its projects had a European dimension. According to the evaluator, this was instrumental in ensuring that this dimension was not lost along the way. The “City and Citizens” dimension in Leeuwarden It is the view of the evaluator that by far the most unique and key principle of the Leeuwarden-Friesland 2018 was linked to the “City and Citizens” dimension, reflecting the ECOC mienskip concept. Engaging with and encouraging local people to develop and implement projects for the cultural programme was a key defining aspect of the ECOC and something that those in the LF2018 Foundation were totally committed to delivering. The final report of the contract supporting the evaluation concludes that this engagement attitude was the central spine of the whole project rather than just a specific and separate community ‘add-on’, which was sometimes the case with past ECOC. Critically, those responsible for overseeing the ECOC in the Foundation saw the citizens of Leeuwarden-Friesland as the producers of the ECOC rather than simply the consumers of it (i.e. were part of the ECOC’s delivery and not just a passive audience). More than 700 of the 800 projects in the cultural programme came from the mienskip approach, where the communities came together to design, plan, often fund and deliver various cultural projects. Three dedicated staff members in the LF2018 Foundation were in place to help develop this open programme, another example on how governance was aligned on declared intentions. Another key impact was that the ECOC stimulated a new wave of amateur performers in the region, who wanted to directly produce rather than just consume culture. Professional artists, performers and technical staff still played a key part in the programme, but a key impact was awakening and stimulating non-professionals to consider performing and participating in culture. Over 60,000 local people took part as performers or volunteers, including teachers, farmers, office workers, restaurant owners, people out of work and schoolchildren as well as cultural professionals. 5.2.Efficiency As was already the case in previous ex post evaluations, it also appears this year from the report of the contractor that overall, the ECOC remains an efficient EU Action, providing good levels of benefits for the EU for relatively little EU investment. Indeed, the only direct contribution from the European Union, in the form of the €1,5m Melina Mercouri prize awarded to each ECOC, is dwarfed by the total amount of money invested by the host cities in designing and delivering the ECOC (the operational budgets were approximately €104.6m for Leeuwarden and € 26.5m for Valletta). Without the initial EU impetus to support the ECOC, it is unlikely that the host cities would have invested anywhere near the amount of funding they did in connection with the ECOC title. The possibility of securing the title typically stimulates cities – but also their respective regional / national public authorities and private partners – to invest much more heavily in culture than they would in the absence of the ECOC. This is true both in terms of infrastructure (this was more the case in Valletta than in Leeuwarden, and that is precisely why the competition started one year ahead of schedule in Malta) and expenditure in cultural events and activities (especially in Leeuwarden). The final report of the contract supporting the evaluation concludes that this means the Action remains highly efficient in terms of returns from the Melina Mercouri prize. As described above, the efficiency of management arrangements varied between the two cities, with those in Leeuwarden-Friesland being generally stronger than in Valletta. Indeed Valletta went through considerable changes and lost important expertise en route, which ultimately had an impact on the artistic direction and content of the cultural programme. Whilst opinions were divergent or even polarised, many key stakeholders (including those involved in implementing the programme and those external to the implementation) expressed the view that the overall artistic direction suffered from the above-mentioned departure of key staff members of the Valletta 2018 Foundation in 2013 and then again in 2017. The political dimension of the Foundation also attracted criticism locally and internationally. Furthermore, as indicated above, funding in Valletta was considerably lower than the one planned in the application, even though it was still sufficient to produce a programme of the scope and scale of a European Capital of Culture as illustrated below.

In Leeuwarden, the arrangements for governance and implementation proved efficient, drawing on the strong political support in the city, the region and the province. According to the evaluator, the ECOC was well run with strong and robust systems, procedures, transparency and leadership. This is reflected firstly in the fact that the Foundation and its partners delivered a cultural programme of the intended scope, scale and quality and raised the necessary resources (the final resources income were even much higher than planned as described above). This conclusion finds another confirmation in that the Foundation generally fulfilled the expectations set for the ECOC, notably in terms of audience engagement, which was at the very core of the mienskip concept. In particular, it appears that there was a genuine link between the ECOC staff and local people, with a high level of trust and openness between the two. This trust and openness manifested itself in relation to an ECOC programme that had over 800 projects spread across a series of social themes and a wide geographic area. The EU-level selection procedure ensured the designation of two ECOC that were relevant to the objectives at EU level and that had the potential to achieve the intended effects. The procedure ensured healthy competition in the Netherlands with five cities submitting strong applications. In Malta, the procedure provided sufficiently rigorous and impartial scrutiny of the Valletta’s ECOC application. However, the Maltese experience suggests that the procedures introduced by Decision 1622/2006/EC might not be appropriate to a situation in which the national Government not only organises the selection process but is also – by necessity – one of the key players in an ECOC application (even if indirectly, e.g. via a body under its control). It is worth noticing that in the meantime, the selection procedure has been changed by Decision 445/2014/EU, providing that the panel is now composed of 10 members designated by EU institutions and bodies (against 7 before) and up to 2 by national authorities (as against 6 before). The EU-level monitoring procedure, with formal and informal monitoring meetings as well as in situ visits with the panel, has proven valuable in giving impartial advice and support to the ECOC from highly experienced experts, some of whom have implemented previous ECOC. 5.3.Effectiveness Ultimately, the effectiveness of the ECOC in achieving the objectives set for them (at EU level and local level) can only be determined in the long-run. That being said, the final report of the contract supporting the evaluation shows that the two ECOC 2018 presented cultural programmes that were more extensive, diverse, innovative and international compared to the baseline cultural offering in previous years. In total, the ECOC cultural programme in Leeuwarden-Friesland was ambitious in its scale and focus, with accent put on the social dimension. It included 800 projects of various sizes and formats, representing a whole range of cultural disciplines, with most of them having a strong social component. As already mentioned, and in line with the mienskip approach, 700 of these projects were bottom-up and emerged from local people, some of them implementing a cultural project for the first time in their life. One can therefore conclude that an overwhelming majority of the projects in the final ECOC programme were genuinely additional. In Valletta, the programme featured 168 projects involving nearly 500 events. Whilst some parts of the programme represented a continuation of existing events (e.g. some festivals), some important projects were new, notably big events such as the opening ceremony, the Valletta Pageant of the Seas, the closing ceremony or key events curated by international artists. A higher number of European and other international artists were involved in both ECOC compared to previous years (see paragraphs on the European dimension above), some new works were created and performed or exhibited and spaces in both cities were used in new ways to host cultural events. As described below, both ECOC helped widen access to and participation in culture although the evidence is stronger in Leeuwarden than in Valletta. Both ECOC also helped strengthen the cultural capacity of their cultural and creative sectors as well as their links with other sectors. Finally, both ECOC widened access to and participation in culture during 2018, although Valletta suffered considerable adverse international publicity. Effectiveness in delivering a cultural programme with a European dimension The Leeuwarden-Friesland cultural programme was of the scale and quality proposed in the original application and had a genuine European dimension. According to the evaluation, the ECOC project has been successful in contributing to making the cultural offering of Leeuwarden-Friesland more European. Indeed, as already illustrated above, the cultural programme featured performances by a diversity of international artists and exhibitions of international works, strengthened European networks and connections, European co-productions and residencies by European artists. The performance of Valletta was also relatively good in this respect. It appears that the vast majority (92%) of respondents responsible for individual projects viewed their work as relevant to European audiences, while 69% believed their project could be exported in the future. Reflecting the European dimension, data from a large-scale cross-sectional survey undertaken within the Valletta 2018 research programme also shows that the sense of attachment to Europe amongst residents of Malta increased significantly, with 73% feeling “very attached” in 2018 compared to 34% in 2016. Effectiveness in reaching and engaging with local citizens, in targeting specific groups and in strengthening the local cultural and creative sectors Both ECOC widened access to and participation in culture during 2018, although the evidence is stronger in Leeuwarden than in Valletta. In Leeuwarden-Friesland, audience figures from the local research showed that 5.4 million people attended ECOC projects up to and including 2018, a figure which increased by 5% each year from 2015 onwards. Some 51% of the audiences were people living in Friesland with a further 42% coming from the rest of the Netherlands. Significant parts of the local population participated in or attended the larger events, with 71% of all 18-84 years olds attending the flagship parade of Royal de Luxe Giants and 32% of this age group attending the Escher exhibition, which was of the flagship exhibitions of the year. Local research showed that 30% of people attending events had little previous affinity to culture meaning a broader range of participation was achieved over the year. Perhaps equally impressive is that around 10% of the total population of the entire region (around 62,000 people) actually participated in ECOC activities either in terms of being a volunteer or delivering an ECOC project (i.e. was a producer and not just a consumer). Furthermore, according to a study published by the Friesland Cultural Planning Bureau, 68% of the total population of Friesland felt involved in the ECOC and concluded that access levels into the programme were high. The ECOC enabled Valletta to attract much greater audiences for culture than in previous years, exceeding 400,000 people for the whole of 2018. More than half of the total audience figure was accounted for by events that were new for 2018, with the big open-air events accounting for much of this figure. Valletta 2018 contributed to increased attendance at or participation in cultural events amongst Maltese residents and provided new opportunities for citizens to be involved as creators, performers and audiences, for example by working with international artists in productions of high professional quality. At the same time, the total audience was considerably less than the stated aspiration of 1 million. Finally, Valletta 2018 run an ambitious volunteer programme, Tal-Kultura, which began in 2016. Aggregate data on the total number of volunteers by the end of the title-year was not available at the time of the evaluation. However, before the start of the title-year, the programme had already 400 volunteers registered (equalling 2,500 hours of volunteer time), varying in age, nationality and background, and with tasks ranging from ushering and guiding to archiving and documentation. Both ECOC also helped strengthen the cultural capacity of the local cultural and creative sectors and their links with other sectors. As far as Valletta is concerned, the ECOC contributed to its culture-driven regeneration by serving as a focus for cultural infrastructure investments, helping to stimulate private investments and implementing cultural activities that attracted visitors and increased the city’s vibrancy. Valletta 2018 also helped develop the capacity of Malta’s cultural and creative sectors, not least through the involvement of some 645 cultural operators in the cultural programme. At the same time, it is the view of the evaluator that the experience of 2018 highlighted some gaps and therefore the need to develop further Malta’s cultural infrastructure as well as reinforce capacity building, knowledge, competencies and skills amongst cultural players. Furthermore, some international artists established relationships with local artists that will endure beyond 2018, although the governance challenges and political controversies tended to reduce the willingness of international artists to work in Malta in future. In Leeuwarden-Friesland, capacity was built both in terms of strengthening the skills and abilities of cultural organisations already established in the area and in relation to the ‘new’ cultural players who were involved in the open programme. Helping build the capacity of grass roots organisations or individuals who often possessed very little cultural skills prior to 2018 was particularly driven by the 1.28 million euros that was spent on project support in the open programme. The LF2018 Foundation also developed an online toolkit specifically to help people develop projects by providing a step by step guide on issues including project design, funding availability, partner development, health and safety, marketing, securing permits finding venues and maximising ticket sales. The Foundation also organised six weekly meetings with all projects to bring them together in order to share news, requirements but also to share knowledge, good practice and advice. Effectiveness in reaching out to European and international audiences According to the evaluation, the ECOC raised the international profile of both 2018 title-holding cities.

As far as Leeuwarden-Friesland is concerned, many stakeholders interviewed for this evaluation stated that it would have been far easier to produce a more “conventional” cultural programme with much less community participation and a larger number of “box office” projects that would have easily attracted the attention of international audiences. Against this background, one can see the fact that some 7% of the audiences (around 300,000 people) were international as a success indicator, even more so as the region was not on the typical tourist route for many international visitors to the Netherlands before the ECOC-year. Overnight stays (often seen as a good proxy for international visitors) showed an increase from 0.8 million in 2017 to 2.1 million in 2018. Estimates from the hotel managers suggested around 65% of the ‘extra’ overnight stays seen in 2018 compared to 2017 were from international visitors. It is therefore safe to say that the ECOC raised the international profile of Leeuwarden-Friesland, albeit from a low base. The evidence suggests that the ECOC contributed to an increase in foreign tourists to Valletta. Tourist visits to Valetta increased from 1.61m in 2015 to 1.75m in 2016, 2.04m in 2017 and 2.37 in 2018. Of course, the overall trend of recent years is one of increased tourist visits to the country in general (+12.7% in air passenger numbers from 2017 to 2018) and it is impossible to isolate the impact of the ECOC. However, more tourists reported that they visited Valletta for cultural reasons in 2018 (475,000) than in 2017 (139,000). In this context, it is safe to conclude that the overall Government strategy (of which the ECOC constituted an important part) to regenerate and promote Valletta as a cultural destination is proving successful. Effectiveness in ensuring sustainability The two 2018 ECOC cities offer somewhat different potential for the sustainability of their activities and of improved cultural governance. The legacy plans for Leeuwarden-Friesland were relatively less developed compared to other aspects of delivery at the end of their ECOC year. There was no continuous and smooth transition into 2019 with a clear and immediate published plan for new cultural activity for the region. However, clearer and stronger legacy plans were announced part way through 2019, which gives clearer direction in terms of legacy and succession planning. The legacy issues relate more to people rather than bricks and mortar in that the ECOC was about getting people to participate in society through culture rather than building new cultural facilities and venues. This means the sustainability of the effects are partly up to whether local people continue to be engaged and whether any legacy organisation post-LF2018 continues to stimulate a bottom-up approach. Although some funding is in place and capacity has been greatly built up in 2018, there is still a concern that momentum will be lost and will not automatically spring into life in the coming years. It is the view of the evaluator that many of the positive effects of Valletta 2018 will be sustained and enhanced beyond 2018. The Maltese Government continues to prioritise culture and the culture-driven regeneration of Valletta, as evidenced by the 19% increase in its budget for culture in 2019 compared to 2018. A key factor has been the creation of the Valletta Cultural Agency “to carry forward the legacy created by Valletta 2018”, which will employ several staff members of the Foundation and operate from the same premises at the centre of Valletta. 5.4.Coherence The ECOC Action is coherent and complementary to the Creative Europe Programme in that it promotes the objectives of Creative Europe and is distinct from the other activities supported by the programme. Leeuwarden and Valletta also made use of funding from Creative Europe to support projects in their cultural programmes. Examples include “Orfeo & Majnun”, which was an interdisciplinary, participatory music-theatre project featuring partners from Malta, Austria, Belgium, France, Netherlands, Poland and Portugal. The ECOC Action is also coherent with and complementary to the European Structural and Investment Funds, depending on the context of each city holding the title. This was most notably the case in Valletta, where some significant investments were implemented in cultural infrastructure receiving co-financing from the ERDF, such as the new MUŻA (which opened the end of the ECOC-year) and the new Valletta Design Cluster (which is a key element of the year’s legacy). Those investments thus supported the successful implementation of the ECOC cultural programme, whilst the ECOC helped “valorise” the investments by ensuring that the venues were used and given greater profile. In Valletta, Interreg also supported Design4Innovation, a collaborative project organised by eight European partners dedicated to promoting design as a tool for user- centred innovation. In line with the aims of the 2018 European Year of Cultural Heritage (EYCH), the programmes of both Valletta and Leeuwarden-Friesland included projects promoting cultural tangible and intangible heritage. As an example, five projects that had been awarded the EYCH 2018 label were presented in the “European Project Slam” during the annual conference of the Network of European Museum Organisations, which was held in Valletta during the title-year. 5.5.EU added value As illustrated above, the ECOC Action achieved an impact that would not have arisen through the actions of Member States alone. The designation of Leeuwarden and Valletta as ECOC attracted benefits that would have been unlikely to arise to the same extent in the absence of the Action. Indeed, the evidence presented in the chapters above suggest that the ECOC, as an EU initiative and an EU branding, provides the stimulus for stakeholders to commit resources and effort to a shared vision and collaborative programme at a much greater scale than would otherwise happen. The EU value added consists largely in the force of the 'brand' itself to act as a significant generator of interest from stakeholders not only from the city and but also from far beyond. This is evidenced, for example, by the creation of dedicated organisations to implement the ECOC and the allocation of resources that are additional to the “mainstream” funding for cultural activities in the cities and regions concerned. For Leeuwarden-Friesland, it is the view of the evaluator that the EU added-value is also to be seen in relation to the ECOC’s very special profile, in that Leeuwarden-Friesland was successful in stimulating over 60,000 local people to become involved in the delivery of the programme in a way that no national or local ‘brand’ never achieved before. Another element of this value added is connected to the need to have a European dimension, which many saw as a strong driver to force the (cultural) community of Friesland to become more outward-looking and move away from its local comfort zones. In Valletta, the award of the title provided impetus to the wider culture-driven generation of the city, whilst the fixed timescale of the title-year provided greater focus than might otherwise have been the case to move forward in that direction. While the national Government’s commitment to culture means that an increase in investment in venues and events may have taken place anyway, the opportunity of the title-year led to those investments being of greater scale and ambition than would otherwise have been the case. Regarding the visibility of the EU, both title-holders gave full prominence to the title of “European Capital of Culture” in their communication and promotional materials and gave prominence to the fact that the ECOC is an EU Action. The results consistently achieved by the Action through the years establish that its models and mechanisms – for example in terms of building a solid governance for the delivery of the ECOC, increasing the capacity of local cultural organizations or attracting projects' ideas from local residents – can be applied not only within one individual Member State that has its administrative and cultural specificities but in any Member States, finally confirming its EU added value. 6.Conclusions The Commission concludes that the ex post evaluation of the two ECOC 2018 confirms what already emerged from all previous eleven yearly ECOC evaluation exercises, i.e. that the ECOC Action is highly valued by the hosting cities that can obtain positive impacts during the development and title years. The action also remains relevant at EU level.

The Commission also concludes that the programmes implemented by the two 2018 title-holders were consistent with the objectives of the ECOC Action: -They reflected its European dimension: as an example, the programme in Leeuwarden-Friesland featured overall 1,600 international collaborations covering 87 countries while Valletta 2018 had a multi-project strand dedicated to “Exile & Conflict” (finding a strong resonance across Europe) and developed an international residency programme of a scale unprecedented in the country; -They involved local residents and stakeholders: in Leeuwarden-Friesland, over 60,000 local people took an active part in the delivery of the LF2018 programme and there was an impressive number of 16,159 registered volunteers whilst Valletta developed a dedicated schools programme that involved around 28,500 children in creative projects over the period 2016-2018; -They widened access to and participation in culture during 2018: audience was at the core of the mienskip concept in Leeuwarden-Friesland and a huge part of the programme (700 out of 800 projects) was developed by – and for – local people; in Valletta, audiences reached 400,000 people (which was less than expected) but the ECOC was successful in providing new opportunities for citizens to be involved as creators, performers and audiences; -They also helped strengthen the cultural capacity of the local community and/or cultural and creative sectors and their links with other sectors: for example, the LF2018 Foundation developed a specific online toolkit to help people develop cultural projects whilst the ECOC contributed to Valletta’s culture-driven regeneration by serving as a focus for cultural infrastructure investments; -The ECOC raised the international profile of Leeuwarden (albeit from a low base) with 7% of visitors coming from abroad whilst Valletta 2018 constituted an important element in the overall Government’s strategy to regenerate and promote Valletta as a cultural destination world-wide; -Both ECOC-years may lead to legacies both physical (in the form of new or refurbished cultural and logistic infrastructures, as it happened mainly in Valletta) and intangible (for example through the new skills developed by the high number of local people who actively developed cultural projects in the case of Leeuwarden-Friesland). However, as was the case with all previous ex post evaluations of the ECOC Action, the current evaluation – because it comes too early after the implementation of the ECOC year in accordance with Article 12 of Decision No 1622/2006/EC – cannot assess the long-term impacts of the two ECOC 2018. As already mentioned in the Staff Working Document accompanying its report on the ECOC 2018 , with the view to addressing this shortcoming, the Commission intends to analyse such impacts in the frame of the evaluation exercise foreseen in Article 16 of Decision No 2014/445/EU. This exercice will build on the results of the study published by the European Parliament in December 2017 on "European Capitals of Culture: Success Strategies and Long-Term Effects". Annex 1: Procedural information 1.Lead DG, Decide Planning/CWP references The evaluation was led by Directorate General for Education, Youth, Sport and Culture (DG EAC). It is included in the Work Programme of Creative Europe for 2018 and in the Agenda Planning with the reference PLAN/2018/2773. 2.Organisation and timing The evaluation was supported by an external and independent evaluator, under a service contract. The service contract was implemented via a Framework Contract with reopening of competition and in accordance to the Financial Rules Applicable to the General Budget of the Union and its Rules of Application . The evaluation roadmap was adopted on 11 April 2018 . According to the roadmap, a Steering Committee including staff from the European Commission’s Directorate General for Education, Youth, Sport and Culture and Secretariat General was established in May 2018. The Steering Committee met in four occasions: to prepare the Terms of Reference in May, to kick off the evaluation in July 2018, to approve the Inception Report in November 2018 and to discuss the draft final report in July 2019. Extensive correspondence between the Steering Committee members was held in between the meetings to follow-up on the evaluation. 3.Exception to the better regulation guidelines The evaluation was exempted from the obligation to carry out an open public consultation. As explained in the Roadmap, the ECOC Action is considered to be a local action. International participation is scattered within and outside Europe and is difficult to reach. A wider evaluation exercise will be carried out at a later stage, with the objective to establish long-term impacts of the initiative as foreseen under Article 16 of Decision 445/2014/EU, which governs the ECOC Action for the titles 2020-2033. The first evaluation of this kind shall be finalised by 31 December 2024. A public consultation will be carried out in this framework in 2023-2024 to gather widest possible views. 4.Evidence, sources and quality The 2018 evaluation of the ECOC used a series of data sets to inform its findings. The main ones being: -Interviews with over 70 stakeholders directly and indirectly involved in the planning, operation and delivery of the two ECOC programmes;

-A literature review of ECOC and European level information on the two ECOC including application/bid information, EC Committee reports, cultural programme brochures, web sites and news articles. The ECOC's own external evaluations have also been used to inform the European evaluation process; -Together, the above evidence base provides the evaluation with a valid and rounded set of data to inform the views on the main aspects of the ECOC evaluation including efficiency, effectiveness, sustainability and relevance. This view has been informed by: ·The scale of the consultation exercise. Around 120 individuals have fed their views and opinions into the evaluation process either through face to face interviews or telephone interviews; ·The nature of the consultation exercise. The evaluators were keen to consult with those who had a more indirect and external view of the two ECOC. These stakeholders including journalists, those not directly benefitting from the ECOC (e.g. rejected projects) as well as those working in the wider cultural policy agenda at regional and city level. This ensures the evaluation is not simply based on those who benefitted the most from the ECOC.



Annex 2: Stakeholder consultation The stakeholders were consulted via targeted consultations (face-to-face interviews or phone interviews). The consultations included the team responsible for the implementation of the ECOC, the political stakeholders involved in the project, the projects participating in the programme or having submitted proposals to participate that were rejected, as well as personalities attached to the cultural tissue in both cities and stakeholders at EU level. The objective of the consultations was to have evidence supporting the findings and conclusions of the evaluation. The interview lists in Leeuwarden-Friesland and Valletta, as well as the topic guides for the various types of interviews can be found in the Annexes 1 and 2 of the report produced by the experts assisting the Commission. The list of stakeholders consulted was partly developed by the ECOC delivery teams but also through an internet search for relevant stakeholders. For the reasons already mentioned in Chapter 4 of the Staff Working Document and in Annex 1, no open public consultation was conducted in the framework of the evaluation exercise. The consultation was restricted to relevant stakeholders in the two cities hosting the title. The characteristics of the Action and the scope of the evaluation do not make it necessary to extend the consultation to a wider public (as indicated in the roadmap published). Annex 3 details the evaluation questions for which the stakeholders' consultation was used. These regard mainly the questions which answers are not based on factual data. The stakeholders' consultation was particularly useful to find information about the impact of the ECOC in the cultural offer of the city, the participation of citizens and local cultural operators, the building of capacity for local cultural operators and legacy prospects. Annex 3: Methods and analytical models The figure below presents the overview of the methodology. A more detailed overview of the methodology and sources used for each of the evaluation questions is presented in the subsequent tables and text. 1. Overview of methodology and tasks of the evaluation: Inception phase Task 1: Kick-off meeting Task 2: Initial consultations Task 3: Desk research Task 4: Inception Report Main Research phase

Task 5: First visits to the ECOC cities Task 6: Second visits to the ECOC cities Task 7: Additional interviews Analysis and reporting phase Task 8: Interim Report Task 9: Factual check of the city reports Task 10: Final report Table 1.1 Evaluation questions: Relevance, EU added value and coherence Evaluation Question Literature reviews Quantitative Data analysis Interviews with delivery teams Stakeholder consultations Survey of projects Project interviews Analysis EQ1: To what extent are the objectives of the ECOC Action? Y Y Y Y Y What was the main motivation behind the city bidding to become a European Capital of Culture? Y Y Y Y Y What was the process of determining objectives? Was there a process of consultation in each city to define aims and objectives? Y Y Y Y What were the objectives of the city in being ECOC? What was the relative importance of each objective? Y Y Y Y Y Y To what extent were the objectives consistent with the Decision and with the ECOC's own application? (special focus on the European dimension) Y Y Y Y Have any specific objectives of the ECOC event been related to social impacts? Y Y Y Y Y In this connection, did the objectives of the ECOC event include reaching out to all groups of society, including the excluded, disadvantaged, disabled people and minorities? Y Y Y Y Y Y EQ2: To what extent were the ECOC's cultural programmes and associated activities relevant to their own objectives? Y Y Y Y Y Y Y To what extent were the activities consistent with the ECOC's own objectives? (special focus on the European dimension) Y Y Y Y Y To what extent have the specific themes/orientations of the cultural programme proved to be relevant to the objectives defined? Y Y Y Y How was the European dimension reflected by the themes put forward by the ECOC event and in terms of cooperation at European level? How did the Capitals of Culture seek to make the European dimension visible? To what extent did the two ECOC cooperate? Y Y Y Y Y Table 1.2 Evaluation questions: Effectiveness Evaluation Question Literature reviews Quantitative Data analysis Interviews with delivery teams Stakeholder consultations Survey of projects Project interviews Analysis EQ3: To what extent were the EU-level objectives achieved? Y Y Y Y Y Y Y Provide typology of outputs, results and possible impacts of the Action at different levels (European, national, regional etc.) Y Y Y Y Y To what extent has the ECOC event been successful in attaining the objectives of the Action (refer to list in the intervention logic)? Y Y Y Y Was the cultural programme perceived as being of high artistic quality? To what extent did the ECOC prove successful in bringing their chosen artistic themes/orientations to the fore? Y Y Y Y Y To what extent did the ECOC title contribute to an enhanced cultural offer in the cities holding the title (e.g. in terms of scope and scale) with stronger European dimension? Y Y Y Y To what extent did the ECOC implementation widen access to and participation in culture in the two cities? What actions were taking to include the elderly, young people, people with special needs in the cultural activities? How accessible were the activities carried out? Y Y Y How did the ECOC programmes help strengthening the capacity of the cultural and creative sectors and its links with other sectors? Which help was available to cultural operators to extend their networks and work transnationally and internationally? Y Y Y Y Y Y To what extent did the Action in the two cities raise their international profile through culture? Y Y Y Y Y EQ4: To what extent were the cities's own objectives achieved? Y Y Y Y Y Y What quantitative indicators (number of visitors, overnight stays, cultural participation of people, etc.) of the social, tourist and broader economic impacts of the event have been gathered by the ECOC? Y Y Y Y To what extent did the ECOC achieve the outputs hoped for by the city and as set out in the application? Y Y Y Y Y To what extent have specific objectives related to social impacts been met? Y Y Y Y Y To what extent were the objectives related to reaching out to all groups of society, including the excluded, disadvantaged, disabled and minorities, met? Y Y Y Y Y Y EQ5: To what extent has the Action resulted in unintended effects? Y Y Y Y Are there any instances where the ECOC event has exceeded initial expectations? What positive effects has this had? Y Y Y Y Y Where expectations have not been met, what factors have hindered the development of the Action? Y Y Y Y Y Have any other unintended effects been identified? Y Y Y Y Y EQ6: To what extent can the positive effects of the ECOC Action be considered to be sustainable? Y Y Y Y Y Which of the activities or elements of the ECOC event are likely to continue and in which form once the ECOC-year is over? Y Y Y Y Y Has any provision been made to continue and follow up the cultural programme of the ECOC event after the closure? Y Y Y Y Y How will the city continue to manage its long-term cultural development following the ECOC event? Y Y Y Y Y What will be the role of the operational structure after the end of the ECOC event and how will the organizational structure change? Y Y Y What has been the contribution of the ECOC event to improved management of cultural development in the city? (in the medium-term) Y Y Y Y Y What are the impacts of the ECOC event likely to be on the long term cultural development of the city? Y Y Y Y Y What are the impacts of the ECOC event likely to be on the long term social development of the city? Y Y Y Y Y What are the impacts of the ECOC event likely to be on the long term urban and broader economic development of the city? Y Y Y Y Y Table 1.3 Evaluation questions: Efficiency EQ7: How did the management arrangements of each ECOC contribute to the achievement of outputs, results and impacts? Literature reviews Quantitative Data analysis Interviews with delivery teams Stakeholder consultations Survey of projects Project interviews Analysis How have the organizational models of the formal governing Board and operational structures played a role in the European Capital of Culture? What role have the Board and operational structures played in the ECOC event's implementation? At what stage were these structures established? How did it improve management of culture in the city during the event? Y Y Y Who chaired the Board and what was his/her experience? What were the key success and failure elements related to the work of the Board and operational structure used and personnel involved? Y Y Y Has an artistic director been included into the operational structure and how was he/she appointed? What were the key success and failure elements related to the work of the artistic director and personnel involved? Y Y Y What was the process of designing the programme? Y Y Y Y Y How were activities selected and implemented? Y Y Y Y Y How did the delivery mechanism contribute to the achievement of outputs? Y Y Y To what extent has the communication and promotion strategy been successful in/contributed to the promotion of city image/profile, promotion of the ECOC event, awareness-raising of the European dimension, promotion of all events and attractions in the city? Y Y Y Y Y To what extent has the communication and promotion strategy including the use of social media successfully reached the communication's target groups at local, regional, national, European and international levels? Y Y Y Evaluation Question Literature review Quantitative Data analysis Interviews with delivery teams Stakeholder consultations Survey of projects Project interviews Analysis EQ8: To what extent were the selection, monitoring and EU co-financing procedures, introduced by Decision 2006/1622/2006/EC efficient? L To what extent have the mechanisms applied by the Commission in line with Decision 2006/1622/EC for the selection of the European Capitals of Culture and the subsequent implementation and monitoring mechanisms influenced the results of the ECOC event? Y Y To what extent has the informal meeting following the designation as well as other advice offered by the panel and by the Commission influenced the results of the ECOC event? Y Y Y How was the Melina Mercouri Prize used? Y EQ9: To what extent did the ECOC manage to raise the necessary resources? Y Y Y Y Y What was the process of securing the financial inputs? Y Y Y Y What was the total amount of resources used for each ECOC event? What was the final financial outturn of the year? Y Y Y What were the sources of financing and the respective importance of their contribution to the total? How much came from the European Union Structural Funds (e.g. ERDF - European Regional Development Fund, ESF – European Social Fund) or other sources of EU funding? Y Y Y Y To what extent did the ECOC title trigger complementary sponsorship? Y Y Y What was the total expenditure strictly for the implementation of the cultural programme of the year (operational expenditure)? What was the proportion of the operational expenditure in the total expenditure for the ECOC event? Y Y Y What proportion of expenditure was used for infrastructure (cultural and tourism infrastructure, including renovation)? Y Y Y EQ10: To what extent were the financial and human resources secured by each ECOC appropriate and proportionate? Y Y Y Y Was the total size of the budget sufficient for reaching a critical mass in terms of impacts? Could the same results have been achieved with less funding? Could the same results have been achieved if the structure of resources and their respective importance was different? Y Y Y Y To what extent have the human resources deployed for preparation and implementation of the ECOC event been commensurate with its intended outputs and outcomes? Y Y Y As a result, could the total budget for the ECOC event be considered appropriate and proportional to what the each ECOC set out to achieve? Y Y Table 1.4 Evaluation Questions: Coherence

Evaluation Question Literature review Quantitative Data analysis Interviews with delivery teams Stakeholder consultations Survey of projects Project interviews Analysis EQ11: To what extent were the ECOC complementary to other EU initiatives? Y Y Y Y Y As far as the conclusions made for the two cities allows it, to what extent has the Action proved to be complementary to other EU initiatives in the field of culture? Y Y Y Y Y To what extent has each ECOC been reinforced by and added impetus to investments by the EU Structural Funds? Y Y Y Y To what extent have the two ECOC complemented other EU initiatives, e.g. European Year of Cultural Heritage? Y Y Y Y EQ12: What is the EU added value and the visibility of the ECOC Action? Y Y Y Y Y Y As far as the conclusions made for the two cities allow, what is the added value of the European Capital of Culture being an EU initiative, compared to what could be achieved if the Action was a purely national or local action? Y Y Y Y Y Y Was the fact that this is a European Union action sufficiently communicated by the cities? Y Y Y Y Y Y To what extent were the general public and the cultural operators aware of this fact? Y Y Y Y Y Y Note: Project surveys were conducted directly by the two ECOC cities. 2.Details of the relevant evaluation tasks and methodologies: Inception phase The aim of the inception phase was to finalise the evaluation framework and research tools, address the project steering group's comments regarding the proposed method and collect background information on the two 2018 host cities. Initial consultations The task consisted in talking to each of the two ECOC, getting evidence about the efficiency and effectiveness of processes at EU level, obtaining qualitative information on key issues for each ECOC to inform later research, identifying additional sources of data and additional interviewees at EU, local and national levels, as well as informing proposals on monitoring and evaluation arrangements. Discussion at the kick off meeting identified a need for strong triangulation and to gain opinions from a broader set of stakeholders. Desk research This involved collection and review of literature, primary data from the delivery agency and available secondary data, as well as web and social media statistics in order to develop a descriptive picture of each ECOC and serve as a source of evidence for later analysis. The list of the literature that was collected at the European level which was reviewed and assessed for usefulness for the sake of the evaluation can be found in the Annex 3 of the Report . Much of the European level literature is helpful to either understand the policy drivers at EU level which support the two ECOC 2018 or provide background on the bidding and application stages for both 2018 cities (i.e. little of it provides information useful for the content, delivery and impact of the two programmes). More academic literature at the EU level linked to the ECOC was also searched for. To complement the above desk research task the contractor assessed the extent to which each ECOC had used big data as well as analysed web and social media to increase visibility and interest in the ECOC among country residents and internationally. There was however no evidence of big data having been used by the Foundation or any of the other stakeholders in Leeuwarden-Friesland and Valletta First visits to the cities Initial face to face visits were made, to Valletta and to Leeuwarden in late 2018. These interviews sought to gather an in-depth understanding of the effect that ECOC had for the individuals involved, their organizations and the local culture sector as a whole. It also allowed to ensure that findings are based on consensus across a range of target groups as well as to identify key differences in the experience of different stakeholders. In the case of projects, the criteria for inclusion focused on whether it represents good practice in relation to at least one of the key dimensions of the evaluation, including (but not limited to) increased European cooperation, the effective targeting of key groups of citizens or neighbourhoods, audience development strategies, or legacy effects. Second visits to the cities With a clear view emerging from desk research as to the overall relevance, efficiency and effectiveness of each ECOC, the contractor undertook a second three day visit to each city in spring 2019. The visit focused on conducting in-depth interviews with the managing teams, their key stakeholders and cultural operators identified during the initial visits. Further interviews were conducted: -with national and local media representatives in order to assess media strategies and impact on communication efforts; -representatives of local commerce, including businesses in the tourism, hospitality and transport sectors as well as key commercial sponsors of ECOC; and -other national ministries or regional authorities to explore regional development issues. Building on the interviews undertaken in the first visits, the interviews with the delivery teams served four purposes: i) gathering further factual data and information about the cultural programme and its achievements; ii) identifying the "story" of the ECOC throughout its lifecycle, i.e. conception, application, development, delivery, legacy; iii) gaining a critical (albeit "insider") perspective on the relevance, efficiency, effectiveness and sustainability of the ECOC; iv) triangulating emerging findings, for example, those emerging from the desk research.