Table of contents 1. INTRODUCTION 3 2. BACKGROUND OF THE INTERVENTION 5 3. IMPLEMENTATION / STATE OF PLAY 10 4. METHOD 14 5. ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 16 6. CONCLUSIONS 47 7. ANNEXES……………………………………………… …………….…………. 50 Glossary Term or acronym Meaning or definition AI /AR/VR Emerging technologies such as artificial intelligence/augmented reality/virtual reality CC Creative commons, a set of copyright licenses developed and offered free of charge by the American non-profit organisation of the same name. CCI Cultural and creative industries CHIs Cultural heritage institutions. This concept covers film institutes, audiovisual / broadcasting archives, institutions for performing arts, national libraries, institutions for monument care, museums of natural history or natural science, museums of ethnography or anthropology, museums of science or technology, national archives, special or other types of libraries, higher education libraries, museums of art, public libraries, archive / records offices, museums of archaeology or history. Cultural material/cultural assets Print (books, journals and newspapers), photographs, museum objects, archival documents, sound and audiovisual material, monuments and archaeological sites DCHE Commission Expert Group on Digital Cultural Heritage and Europeana. The group is composed of representatives of the Member States and observers from Member States, third countries, candidate countries and European/international organisations. A sub-group has been set up and has 12 members. More details are provided by the Register of the Expert Groups . EEA European Economic Area Europeana (or Europeana initiative) Europeana is an initiative, built through a series of EU-funded projects, that has created a platform for unique access to Europe’s digitised cultural heritage resources, developed common standards and solutions to achieve data interoperability and accessibility, and nurtured cross-border visibility and use of cultural resources. Europeana network (or ecosystem) The network of cultural heritage institutions, domain and national aggregators and professionals contributing to Europeana. GLAM Galleries, libraries, archives and museums ISG Interservice Group Metadata Set of text data that describes the cultural objects. PPP(s) Public- private partnership(s) the Recommendation Commission Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU) SWD Staff working document 1.Introduction 1.1. Purpose and scope In an age of rapid technological progress, digital technologies bring unprecedented opportunities to the cultural heritage sector. They provide more effective tools to digitise cultural heritage assets, for diverse purposes such as preservation, conservation, restoration, reproduction, research, education, as well as for a broader, more democratic online access and reuse in key ecosystems such as sustainable tourism and the cultural and creative industries. They provide the public with numerous ways to access, discover, explore and enjoy cultural material while cultural heritage institutions can reach broader audiences, and engage them in innovative ways by offering immersive, creative and accessible content.

Cultural heritage is an important contributor to the European economy, fostering innovation, creativity and economic growth. For example, cultural tourism accounts for 40% of all tourism in Europe, and cultural heritage is an essential part of cultural tourism. There are places, regions and even some countries where the local economy relies extensively or even completely on the revenue that cultural heritage sites and venues generate by attracting visitors to those areas. Digitisation of cultural heritage and the reuse of such content can generate new jobs not only in the cultural heritage sector, but also in other key areas such as the creative industries (e.g. the video game industry, film industry). Such processes also provide incentives for companies to develop more efficient technologies, and new business models. However, cultural heritage, including monuments, archaeological landscapes, buildings and artefacts, continues to face significant threats of deterioration or partial or full destruction or disappearance. Some threats are human-derived, e.g. mass tourism, terrorism and vandalism, while others originate in nature, e.g. deterioration over time, pollution, flooding, wind erosion. There have been regular, tragic reminders (e.g. fires at the Nantes Cathedral in 2020, at the Notre Dame in Paris in 2019, at the Cavallerizza Reale in Turin in 2019, the theft from the Green Vault at Dresden’s Royal Palace in 2019) of the need to leverage digital technologies to avoid the complete loss of the memory of our cultural heritage assets and to enable their reconstruction and reproduction. The unprecedented challenges of the COVID-19 pandemic, with its significant social and economic consequences, have heightened even more the need for and the urgency of using digital technologies. Many cultural heritage institutions (‘CHIs’) had to close and thus incurred major financial loss. However, despite the loss of income, reaching 75-80% for some CHIs, many of them have managed to bounce back and keep or extend their audiences. There has been an increase in the digital services they offer (e.g. engaging with audiences, sharing collections, offering digital tools), demonstrating once more their high value to society. The Commission Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU) (“the Recommendation”), is the Commission’s main policy tool for digital cultural heritage. Its aim is to optimise the use of information and communication technologies to unlock the full economic and cultural potential of Europe’s cultural heritage. The Recommendation dates back to October 2011. Some of the challenges facing the cultural heritage sector at that time, i.e. an urgent need to protect and preserve European cultural heritage at risk are still present today, but others have arisen since. Major advances in digital technologies for example have created new opportunities that did not exist when the Recommendation was adopted. Therefore, the Recommendation needs to be assessed in order to decide whether or not it still meets the needs and expectations of the cultural heritage sector, of the users, of the creative sectors, of society in general. In this context of major digital developments, a clear, evidence-based assessment of the Recommendation will help the Commission to decide on the follow-up steps, such as a revision of the Recommendation. In line with the Better Regulation Guidelines , this evaluation assesses how the Recommendation has been working and actually performing compared to initial expectations by examining its effectiveness, efficiency, relevance, coherence and EU added value. This staff working document (‘SWD’) addresses the five areas in the Recommendation, i.e. organisation and funding of digitisation; digitisation and online accessibility of public domain material; digitisation and accessibility of in-copyright material; Europeana; digital preservation, taking into account the digital lifecycle approach to cultural material. This evaluation is supported by an externally contracted study (’evaluation study’), by various meetings and workshops with Member States, stakeholders and experts in the field, and by targeted and online public consultations. The evaluation covers the whole period since the Recommendation’s adoption in 2011 to date. It covers the Member States, the European Economic Area countries (‘EEA’), Norway, Iceland, Lichtenstein, as well as the UK. 2.Background of the intervention 2.1. Description of the intervention and its objectives The legal basis for the Recommendation is Article 292 of the Treaty on the Functioning of the European Union, which lays down powers for the European Commission to adopt recommendations in specific cases provided for by the Treaties. At the time of its adoption, the Recommendation was an important action for one of the key areas tackled by the Digital Agenda for Europe, a flagship initiative in the Europe 2020 strategy of March 2010. It built on the work done through the digital libraries initiative, with a view to making Europe’s cultural and scientific heritage accessible to all online. The launch of Europeana in 2008, Europe’s digital platform for cultural heritage, was one of the most important stepping stones for digital cultural heritage. The Recommendation built on the 2006 Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2006/585/EC) , which had been an important means of supporting and encouraging the work of the Member States and CHIs in the field. The Recommendation reflected the necessary changes in the policy context and in the cultural heritage sector since 2006, such as the establishment of Europeana, the recommendations stemming from 'the New Renaissance' report, prepared in 2011 by the Comité des Sages, and the European Commission's proposal for an Orphan Works Directive in 2011. Moreover, the Member States‘ reports on the implementation of the 2006 Recommendation for the reporting cycle covering 2008 to 2010 showed progress, but this was inconsistent and uneven for the different provisions of the 2006 Recommendation. A modern and more effective set of measures for digitising and bringing cultural heritage online was needed to reflect the emerging new challenges and opportunities for the sector at that time. In particular, the Recommendation sought to address Europe’s untapped economic and cultural potential. Increasing digitisation and developing and applying advanced technologies to the sector simulates growth and job creation, as the process of digitisation is labour intensive and generates new jobs. Digitised cultural heritage resources are long-term economic assets which, through the data attached to them, create value and contribute to economic growth, with spillover effects into tourism, education, research, etc. The Recommendation sought to address the need to better harmonise national approaches across Member States to the digitisation and preservation of cultural heritage. Considering the resources required for digitisation, a concerted action from the Member States was necessary to ensure more coherence in the selection of cultural material, better optimisation in the use of digitisation capacity and achieving economies of scale, as well as a more secure and stimulating climate for companies willing to invest in digitisation technologies. Stimulating innovative ways to make cultural and creative content available across the Member States results in more possibilities for users to explore and enjoy European cultural heritage treasures.

The Recommendation sought to address the need for increased digitisation, online access and preservation to avoid losing our cultural heritage assets and their memory. The increasing natural or human derived threats to historical buildings, museums and other cultural heritage assets also made it clear that, without digitisation and preservation of cultural heritage treasures, society would run the risk of losing its very ‘memory’ (digital and physical) and knowledge in the event of destruction or disappearance of those assets. The ultimate impacts expected from the Recommendation were a better protection of Europe's cultural heritage, higher public awareness and appreciation of Europe's cultural heritage, increased availability of open data and exchange of knowledge. The intervention logic of the EU action, in the form of a diagram summarising how the EU action was originally expected to work at the time of adoption/implementation, is provided below: The Recommendation had the following main overall objectives: ·optimise the cultural and economic potential of Europe’s cultural heritage ·ensure Europe’s position as a leading international player in the field of culture and creative content; and ·unlock investments from Member States and the private sector for digital cultural heritage initiatives The main measures promoted in the Recommendation are: 1.Digitisation-organisation and funding: Member States are recommended to further develop their planning and monitoring of cultural material (targets and overviews), to encourage partnerships between cultural institutions and private sector, make use of the EU’s Structural Funds to co-finance digitisation activities, consider ways to optimise the use of digitisation capacity and achieve economies of scale; 2.Digitisation and online accessibility of public domain material: Member States are recommended to improve the access to and use of digitised cultural material in the public domain; 3. Digitisation and online accessibility of in-copyright material: improve conditions for the digitisation and online accessibility of in-copyright material; 4. Europeana: contribute to its further development; 5.Digital preservation: reinforce national strategies for the long-term preservation of digital material, update action plans implementing the strategies and exchange information with each other on the strategies and action plans; provide in their legislation for multiple copying and migration of digital cultural material for preservation purposes; make the necessary arrangements for the deposit of the material created in digital format. Point 12 of the Recommendation encourages Member States to inform the Commission every 2 years of their implementation actions. The Commission monitors the progress made by the Member States in implementing the Recommendation with the help of the Expert Group on Digital Cultural Heritage and Europeana. 2.2. Baseline and points of comparison 2.2.1. Digitisation: organisation and funding According to the evaluation study, when the Recommendation was adopted, the state of digitisation varied significantly depending on the type of institution and cultural material. While most CHIs had a digital collection or were involved in some digitisation activities (83%), only an estimated 20 % of all collections that needed to be digitised had in fact been digitised. Moreover, only one third of the institutions were part of a national digitisation strategy and only 34 % had a written digitisation strategy. Art museums were the most digitised (at 42%), while national libraries had digitised only 4% of their collections. Photographs were the most commonly digitised type of cultural material, followed by archival material.

Moreover, the Commission’s Second progress report on the digitisation and online accessibility of cultural material and on digital preservation in the European Union from 2010 , noted that, despite the important progress brought about by the 2006 Recommendation, the work carried out at the time to digitise cultural heritage content was still in its early stages. In particular, the report indicated that the vast majority of cultural objects worth digitising had not yet been digitised (e.g. for national libraries, only 3.5 % of relevant cultural objects had been digitised). While the national reports indicated that inventories, overviews and databases were increasingly being set up, to ensure a cross-sectoral and national dimension, several Member States had still not developed any national overviews of digitised collections The Commission’s report pointed to the fact that progress in digitisation activities depended on the long-term financing available, but very often a national budget had not been assigned to it and it was mainly the CHIs’ own resources that were financing digitisation (followed by project based financing with national and EU funds). Likewise, according to the report, some countries provided information about the sum of public budgets aggregated at national level specifically devoted to the digitisation of cultural heritage on an annual basis (e.g. France, Spain and Finland) or multiannual basis (e.g. Czechia and Belgium ). Furthermore, the evaluation study points to the limited adoption of policies on digital access by the institutions (around 31 % of them). Only 42 % of institutions reported that they were monitoring the usage of their digital collections through consultations or social media monitoring (libraries between 56 % and 78 %, museums around 27 %). The monitoring of use of their digital collections through web statistics was higher (~85 %). According to the Commission’s second progress report of 2010, mentioned above, 17 Member States had national coordination mechanisms for digitisation activities. As far as the targets were concerned, although there were quantitative targets in most of the Member States, they were usually set ‘bottom-up’ by the CHIs themselves and there was no comprehensive aggregated planning document at national level. With regard to public private partnerships/PPPs (i.e. between CHIs and the private sector) although theoretically they could be a useful instrument to pool new resources for digitising cultural material, the national reports indicate they were not a widespread tool for financing digitisation. Large-scale PPPs with both public and private partners (such as Google) had less difficulties in emerging than PPPs involving medium and small-scale cultural organisations. 2.2.2. Digitisation and online accessibility of public domain material and of in-copy right material According to the Commission’s second progress report of 2010, only a minority of Member States (4) had mechanisms making it easier for cultural institutions to digitise orphan works and make them accessible online. Likewise, only a minority of countries (8) declared having mechanisms for out-of-print (out-of-commerce) works. Moreover, only three countries indicated having lists of orphan works and works in the public domain. However, several of the countries took part in the EU-funded ARROW project ‘Accessible Registries of Rights Information and Orphan Works towards Europeana’, involving national libraries, publishers and collective management organisations. 2.2.3. Europeana The Commission’s second progress report of 2010 pointed to a steady growth over time of access to digitised content through Europeana, reaching and significantly exceeding the original target of 10 million items by 2010 (i.e. more than 14 million items, including 5 million text items, were accessible through Europeana). However, despite an improvement in the overall balance, many Member States were not contributing a sufficient amount of content (i.e. about half of the Member States contributed in total less than 1 % of the total number of objects). 2.2.4. Digital preservation The Commission’s second progress report of 2010 highlighted that the preservation of digital information remained a significant challenge, despite a growing acknowledgement by Member States of the costs linked to losing digital data. Almost half of them had adopted a strategy for digital preservation. However, the diversity of the national strategies was ‘ranging from a preliminary feasibility study to a much more advanced and comprehensive digital preservation infrastructure with processes already up and running’ . In addition, according to 'the New Renaissance' report of the Comité des Sages, referred to above, while mechanisms and plans had been put in place by several Member States, there was still a lot of work to be done to put all the strategies and plans into practice. This was the case not only in the Member States that were less advanced in this sector (and running the risk of permanent loss of digital content), but also in the Member States that were more advanced in this area. According to the Commission’s 2010 progress report, 22 countries reported allowing multiple copying and migration of digital cultural material by public institutions for preservation purposes. On policies and procedures for the deposit of material originally created in digital format, most Member States reported legal deposit policies in place for born-digital material, although some also reported actual challenges with implementation. 2.2.5. General aspects From a more general perspective, whereas the evaluation study points to considerable room for improving the supply of digitised cultural material, on the demand side it acknowledges that there were few relevant and specific statistics on the use of digitised cultural material at that time. Overall, the baseline shows that, at the time, despite the efforts and the progress of the Member States and CHIs to digitise, preserve and provide online access to digitised cultural heritage content, further steps were needed . 3.Implementation / state of Play 3.1. Description of the current situation The Recommendation represented a milestone in digital cultural policy, encouraging Member States to step up efforts and support CHIs with the digitisation, online access and preservation of cultural heritage. 3.1.1. Digitisation: organisation and funding

As mentioned in Section 2, the Recommendation encourages Member States to further develop their planning and monitoring of cultural material (targets and overviews), foster partnerships between cultural institutions and the private sector, make use of the EU’s Structural Funds to co-finance digitisation activities, optimise the use of digitisation capacity and achieve economies of scale. According to the information provided by Member States in their implementation reports for the reporting period 2015-2017 and consolidated in the Commission progress report of 2018, 23 of them have national digitisation plans and/or national funding programmes, as summarised in the table below. Figure 1: Predominant trends in the organisation of digitisation across the EU National strategies National funding programmes Regional strategies Regional funding programmes Advisory task forces, committees & work groups National institutions/ Domain specific strategies BE √ √ √ √ BG √ √ CZ √ √ √ DK √ DE √ √ √ √ EE √ √ √ IE √ √ EL √ √ ES √ √ √ FR √ √ √ HR √ √ √ IT √ √ CY √ √ √ LV √ √ √ LT √ √ LU √ HU √ √ MT √ √ NL √ √ AT √ √ PL √ √ √ PT √ RO √ √ SI √ √ SK √ √ √ FI √ √ √ √ SE √ √ √ √ UK data not available for the reporting period Source: Consolidated progress report on the implementation of Commission Recommendation, 2015-2017 The digitisation schemes have targets that vary considerably according to the Member State and that are usually set by each domain of cultural heritage (e.g. libraries, archives, museums, etc.). Digitisation of library and archival cultural resources is one of the main digitisation priorities in more than half of the Member States. In addition, for more than one third, immovable cultural heritage (e.g. monuments, historical buildings, archaeological sites) was a priority in the reporting period and these Member States allocated funding programmes to it. At least six Member States reported digitisation programmes for intangible culture (e.g. living arts, traditional folklore culture), while 3D appears to be gaining importance across the Union. However, as recalled also by contributions to the online consultation, only 22% of the heritage collections have been, on average, digitally reproduced. Discussions in the workshop on digital transformation in the cultural heritage sector, held in Lisbon in November 2019, pointed to the fact that the sector struggles to set priorities for digitisation and preservation and needs a clearer expression of the expectations and set benchmarks. Moreover, according to the national reports, public-private partnerships followed a slow, upward trend, with mainly technology companies (e.g. Google) being interested in such schemes, followed by media publishers (e.g. Proquest) and banks. Some Member States (Czechia, Germany, the Netherlands, Austria, Italy) reported PPPs between national or state libraries and Google: in Czechia around 150 000 books had been digitised and made available since 2011 (when the PPP was established), while in Germany the PPP with Google led to the digitisation of around two million books. Wikimedia, international non-profit organisations and private individuals also concluded agreements with CHIs. Two thirds of the Member States reported making use of the EU Structural Funds for digitising of cultural material and related services (Bulgaria, Czechia, Germany, Estonia, Greece, Spain, Croatia, Italy, Cyprus, Latvia, Lithuania, Malta, Poland, Romania, Slovenia, Slovakia, Finland and Sweden). There are various ways to pool digitisation efforts to optimise the use of digitisation capacity and achieve economies of scale, ranging from competence centres and national or cross-border collaborations to sharing services and facilities. Eighteen Member States (Belgium, Bulgaria, Croatia, Cyprus, Czechia, Estonia, Finland, France, Germany, Hungary, Ireland, Latvia, Lithuania, the Netherlands, Poland, Slovakia, Spain, Sweden) reported competence centres, aiming to centralise funds and lower costs, consolidate the digitisation process, share best practices, etc. Moreover, two thirds of Member States reported sharing services and facilities (e.g. repositories, content management systems or IT tools) to store and manage digital objects and data more efficiently and make better quality content available online (Austria, Croatia, Denmark, Estonia, Finland, France, Greece, Italy, Latvia, Lithuania, Luxembourg, Malta, the Netherlands, Poland, Slovenia, Slovakia, Spain and Sweden). A report on digitisation and IPR in European museums, published in July 2020, shows that 43.6% of museum collections are digitised, although with different levels per museum category. According to the participating museums, less than 20% of their collections are available online. 3.1.2 Public domain material and in-copyright material The Recommendation encourages digitisation and online access to public domain and in-copyright material. The national reports pointed to a positive trend among Member States to ensure that public domain status is maintained after digitisation; most of them reported supporting actions ensuring wider access or use of the digitised cultural heritage material in the public domain. Based on the information provided by the Member States in their national reports, the measures laid down in accordance with the Orphan Works Directive do not appear to have contributed to large-scale digitisation of orphan works by CHIs. For the digitisation of out-of-commerce works, although a significant number of Member States reported licensing mechanisms supporting it, their practical application has been limited, due for instance to the fact that only some types of works (e.g. literary works) are concerned. 3.1.3. Development of Europeana The Recommendation set the indicative target of having 30 million digital objects available on Europeana by 2015. The quantitative targets for Europeana were already reached in 2013 and surpassed, (50 033 909 in July 2020), with images (27 945 338) and text (20 819 920) being the main categories of content, compared with a marginal contribution of 3D for instance (16 662). Although the focus of the CHIs appears to be mainly the quantity, quality emerged as a need as well. More than a third of the Member States encourage their CHIs to submit high quality content and metadata to Europeana and 63.08% of the content is in tier 2 and above, as set by Europeana. A minority of Member States (Austria, Czechia, Germany, Greece, Estonia, Poland, Slovakia and Spain) reported making public funding for digitisation projects conditional upon making the digitised material available through Europeana. Twenty Member States reported contributing to Europeana through national aggregators and through cross-border thematic and domain aggregators. Member States highlighted various challenges to aggregation, such as the need to strengthen the coordination capacity of national aggregators and to address the digital capacity of CHIs and more generally their digital transformation: for example by tackling difficulties with data submission and the lack of expertise in copyright legislation. Around half of the Member States reported a wide and free availability of metadata of digitised cultural content. The reports also pointed to the need to further consider emerging technologies that can improve the quality of the digitised material.

Only a limited number of Member States (Austria, Cyprus, Greece, Luxembourg, Poland, Portugal and Spain) reported raising public awareness of Europeana through national communication actions or plans. 3.1.4. Digital preservation. The national reports point to a mixed coordination approach at different levels to long-term preservation strategies and action plans, with most of the Member States reporting mainly strategies at the national level, specific activities by public institutions or bodies, and participation in European projects. Almost half of the Member States and Lichtenstein reported explicit provisions, in their national copyright laws or laws on archiving, which allow multiple copying and migration of digital cultural material by public institutions for preservation purposes. This reporting includes amendments or new developments at the national level (i.e. in Czechia, Estonia, Germany, and Ireland). Most of the Member States reported having in place measures ensuring that rights holders deliver works to legal deposit libraries and allowing web-harvesting techniques, while half of them have adopted legal provisions allowing the transfer of digital legal deposit works among legal deposit libraries. 4.Method 4.1. Short description of methodology: The findings and conclusions of this report are based on a number of sources, including an evaluation study conducted by an external contractor, several consultation activities, desk research, various documents and exchanges with Member States and stakeholders. The following main methodological steps were followed: 1.Inception phase: This process implied organising the process, setting its objectives and the timeline and selecting the tools and the deliverables. In other words, the first phase focused on starting and agreeing on the conditions for the study (i.e. on the objectives and timing), carrying out an initial desk review and developing data collection tools. This step mainly involved the contractor and the Commission (which included the Interservice Group (ISG). 2.Data collection: This process implied establishing the baseline of the analysis, carrying out national research, desk research and stakeholder consultations (including targeted and online public consultations) and organising workshops. This involved the contractor, various parts of the Commission, Member States’ competent authorities, cultural heritage institutions, international and umbrella organisations, the Europeana ecosystem of organisations and professionals, technology stakeholders with research, products and services in ICT for cultural heritage and the public. The publication of a roadmap defining the scope of the evaluation gathered eight responses from various respondents. The evaluation study was conducted by the contractor between August 2019 and March 2020 and updated in November 2020, following the online public consultation. The study was one of the main sources of information for this staff working document. A consultation strategy was created and agreed with the ISG, with the following objectives: (1) to collect views on the Recommendation’s implementation and (2) to collect views on the impacts of possible future changes to the Recommendation. As part of the consultation strategy, the Commission held five workshops with representatives of Member States, stakeholders, international organisations and networks in order to collect relevant and solid information on the Recommendation’s impact and on the challenges and needs confronting the sector, that have arisen from the digital transformation and that have to be addressed in the future. Between November 2019 and January 2020, the Commission held a targeted consultation of Member States’ authorities with questions covering all five of the evaluation criteria. The Commission received 20 responses that fed into the evaluation study. Between 22 June and 14 September 2020, the Commission held a twelve week open public consultation, focusing on the importance of digitisation and digital transformation in the field and how such processes can be supported, as well as on the Recommendation’s impact on the cultural heritage sector. The consultation gathered 565 responses. No significant limitations were encountered. More details are provided in the synopsis report in Annex 2. An additional main source of information for the study and the evaluation were the national implementation reports by the Member States, which were consolidated in the Commission’s progress report (2018) on the Recommendation’s implementation (see Section 2 above).

3.Data analysis: This implied assessing and compiling the results of research, surveys and consultations, evaluating and assessing findings and discussing them progressively at expert workshops. This involved the contractor, as well as the Commission (which included the ISG). 4.Synthesis and conclusions: This implied drawing conclusions from the findings of the evaluation study, considering other inputs from stakeholders, and drawing conclusions on each evaluation criterion. This was mainly carried out by the Commission. The final steps of this phase mainly consist of: ·assessing the outcomes of the evaluation study, of the consultation activities and of the implementation reports and then ·concluding on Recommendation’s impact in the cultural heritage sector across the EU and more particularly on its effectiveness, efficiency, coherence, relevance and EU added-value. A more detailed description is provided in Annex 3. 4.2. Deviations from the evaluation roadmap The evaluation roadmap published in July 2019 indicated that the evaluation would be completed during the third quarter of 2020. However, due to the COVID-19 pandemic and related priorities, the Commission opted to extend the evaluation timeframe until the end of 2020 to allow sufficient time to process and analyse the results. 4.3. Limitations and robustness of findings Quantifying the costs and benefits of the intervention raised significant challenges. This was due, on the one hand, to the diffuse and intangible nature of the benefits of cultural heritage, and on the other, to the lack of available or updated data on costs in Member States. The lack of available or relevant data made it difficult to provide a monetarised analysis of the cost-benefits of the Recommendation and its impacts. For example, for cost data, the most recent figures on the costs of digitisation, date back to 2017. Moreover, the reports do not always provide relevant or consistent data, making it difficult to establish causal links and to draw proper conclusions. However, where quantitative data are missing, alternative proxy data were used and explicitly mentioned in the analysis. The evaluation is considered robust and thorough. 5.Analysis and answers to the evaluation questions The main findings of the evaluation, based on the five evaluation criteria, i.e. effectiveness, efficiency, relevance, coherence, EU added value, are summarised below and a list of the evaluation questions is provided in Annex 4. 5.1 Effectiveness The assessment of effectiveness considers how successful EU action has been in achieving or making progress towards achieving its objectives. It looks particularly at: a) the extent to which Member States have implemented the Recommendation’s provisions, b) the extent to which the Recommendation’s objectives have been achieved and the achievements can be attributed to the EU intervention; what are the factors preventing these objectives from being achieved or that are helping them be achieved. Key points on the baseline situation ü The level of digitisation across the Member States was low and CHIs and a vast majority of cultural objects worth digitising were not digitised. üDespite an increasing trend, many Member States contributed rather little to the content through Europeana.

üPPPs were not a widespread tool for financing digitisation. Key findings on the current situation: üThe Recommendation’s implementation by the Member States varies to a large extent. üThe Recommendation led to i) an overall improvement in digitisation, online access and digital preservation, which was the result for instance of the adoption of national strategies and/or funding programmes for digitisation and domain-specific initiatives; ii) an increased number of PPPs; iii) more visibility and exchanges, leading to increased support and funding for cultural heritage; iv) practical measures to optimise the use of digitisation capacity, with positive trends since 2011, which can be attributed to the Recommendation. üThe volume of digitised cultural heritage resources has increased, with the target levels in Europeana being exceeded ahead of schedule. üThe overall impact of Europeana has been positive and acknowledged by those consulted for this evaluation and for the evaluation of the Europeana initiative the Commission carried out in September 2018. üHowever, the quantity of digitised cultural heritage should not be dissociated from quality; PPPs were concentrated in several countries, with smaller Member Sates experiencing more obstacles; üImportant challenges related to copyright are expected to be addressed by the transposition and implementation of the Directive on copyright and related rights in the Digital Single Market (2019/790/EU) üThe Recommendation needs to be revised in order to address also the factors hindering the achievement of its objectives (e.g. lack of common standards and approaches, insufficient digital literacy and skills, insufficient funding) and to enhance its effectiveness. The Recommendation has been effective overall. However, after 10 years, it has reached a ‘plateau’ in terms of its effectiveness and would benefit from a revision to increase its effectiveness. This is supported by the feedback from the online public consultation, where 41% of the respondents considered that the Recommendation’s provisions have been implemented effectively and have achieved their objective to improve conditions in the areas addressed. As shown in Section 3 above, the extent to which the various provisions of the Recommendation have been implemented varies according to the Member State. This is the case for instance for the planning and monitoring of digitisation. In this respect, there has been a clear overall improvement resulting from i) the adoption of national strategies and/or funding programmes for digitisation, ii) domain-specific initiatives led by ministries, the use of regional schemes, and iv) quantitative targets at national, regional, institutional or programme level. For example, between 2011 and 2013, only 9 Member States reported using national strategies or funding programmes to digitise cultural heritage material, whereas between 2015 and 2017, 23 Member States reporting doing so. This increase can be attributed to the Recommendation. In particular, as acknowledged also during the meeting of the DCHE of November 2019, the Recommendation stimulated discussions on digital for cultural heritage. It also raised awareness at all levels and across various sectors (e.g. finance), increasing the visibility of the domain, its needs and challenges, which led to more concrete support at national level (e.g. for digitisation projects). During the DCHE meeting of November 2019, participants highlighted the effectiveness of the list of indicative targets for minimum content for each Member State’s contribution to Europeana, as laid down in Annex II to the Recommendation. The list represented an important incentive for the CHIs and Member States in their work to increase the quantity of digitised assets. However, stakeholders also pointed out the fact that the Recommendation’s current provisions focus rather on increasing the quantity of the digital cultural material, and this might have triggered unexpected negative consequences for the quality. Achieving a high volume of digitised assets while ensuring high quality remains a challenge that needs to be addressed. A key factor in supporting high-quality digitised cultural material is capacity building, with current capacity not responding sufficiently to the needs and challenges of the sector (see details below). For the private investment through PPPs, the Commission’s consolidated progress report (2018) shows that the number of partnerships and institutions involved in PPPs has been gradually increasing across the EU, with technology companies such as Google and Proquest (for libraries), Family Search International (for archives) and Telefónica (for libraries and museums) continuing to be the main private partners for large-scale digitisation projects. Additional evidence reveals PPP agreements with local software and technology companies in the field of cultural heritage mostly for digitisation, online access and digital preservation purposes. The evaluation study further points to specific challenges in the role and capacity of the PPPs in achieving the objectives of the Recommendation, challenges that were confirmed during the consultation activities. For instance, while smaller Member States, such as Estonia or Luxembourg, had difficulties in defining such partnerships, most PPPs tended to be concentrated in only some bigger Member States, such as France and Italy. Evidence on the Netherlands, Hungary and Estonia confirms that they are limited and, as in the other Member States, predictably with technology companies and media publishers of educational materials. During the workshop held by the Commission in November 2019 in the context of the DCHE, stakeholders questioned the effectiveness of the provisions/conditions for public-private partnerships for digitisation, as laid down in Annex I to the Recommendation. Participants to the workshop on digital transformation in cultural heritage underlined the significant role the private sector has in driving forward innovation and suggested that further collaboration should be sought to support digitisation efforts. Since the Recommendation’s adoption, crowdfunding and match funding have emerged as alternative sources of funding for digital cultural heritage initiatives. In 2016, for example, Film Archive Austria launched a campaign to restore an historical silent film, garnering support from over 700 contributors, who donated over EUR 86 000. In case of European Structural Funds, 18 Member States reported using European Structural Funds to co-finance digitisation of cultural heritage during the 2014-2020 programming period. According to the evaluation study incorporating contributions from the Member States, Structural Funds played a key role in funding the digitisation of Europe’s cultural heritage in those countries where national sources were limited. The targeted consultation suggests that the Recommendation has been effective in helping to raise additional funds for digitisation, for example by giving the cultural heritage sector the opportunity to clearly convey the need for more support from policy makers. When consulted, the representatives of the Member States in the DCHE also emphasised the Recommendation’s effectiveness in i) stimulating certain activities in their countries (e.g. for the development of a national digital library, digitisation projects), ii) prioritising digitisation over other matters and providing financial support, iii) enhancing coordination and drawing up digital strategies and iv) building up infrastructure at the national level. However, stakeholders pointed to the fact that reliable funding sources for CHIs for the long term remain a challenge. Findings from the Member State consultation survey indicates that while the Recommendation has generally been a useful instrument for promoting digitisation initiatives at national level, more work needs to be done to unlock long-term investments from ministries across Europe (see section on Europeana below). The Recommendation encouraged Member States to consider ways to maximise the use of digitisation capacity and achieve economies of scale. Member States reported that practical measures to optimise the use of digitisation capacity since 2011 have been positive, and this can be attributed to the Recommendation. Examples of such measures include the establishment of competence centres, the use of shared services or facilities, and national or cross-border collaborative measures to pool digitisation efforts and resources. One example of innovative cross-border initiatives among CHIs is the Digital Heritage Network (DEN), previously referred to as the Netherlands Coalition for Digital Heritage. Overall, the Recommendation helped to develop closer cross-border cooperation and increase digital capacity in the cultural heritage sector. This has been confirmed by the representatives of the Member States in the DCHE, who acknowledged the Recommendation’s effectiveness in facilitating the participation of CHIs in the Europeana network and its activities, contributing to a new mind set of openness and sharing. Participants in the December 2019 workshop with international organisations and networks of December 2019 underlined that the Recommendation would be more effective if it further strengthened the various networks of CHIs and in particularcollaboration at international level (including by providing more support for the transfer of technology, the development of skills in non-EU countries).

However, feedback from the consultation activities point out that better coordination is still needed to ensure interoperability (including semantic) and standardisation. The Recommendation encouraged Member States to improve access and use of the digitised cultural material that is in the public domain, as well as the conditions for digitisation and online access of in-copyright material. The emergence of GLAM labs demonstrates how CHIs have made public domain material more accessible and reusable online. GLAM labs promote innovative ways of working, bringing together institutions, technology, people and various communities. Examples include the International GLAM Labs Community, established in 2018, as a hub for international collaboration on digitisation and online accessibility. Moreover, at the workshop on digital transformation in cultural heritage in Brussels in January 2020, stakeholders acknowledged that the Recommendation had helped to trigger such initiatives. However, findings from the workshops indicated that the Recommendation does not sufficiently address all the relevant copyright issues hindering the CHIs in promoting online access to and the use/reuse of digitised cultural heritage; CHIs requested an ecosystem aligning the rules of the Member States (i.e. with EU rules properly supported and enforced). Moreover, there is a general expectation that the provisions of the Directive on copyright and related rights in the Digital Single Market will address some of the current obstacles. With regard to the Orphan Works Directive (2012/28/EU), by 2015 all the Member States had transposed the Directive. As pointed out in the evaluation study, Member States have promoted the availability of databases with rights information, connected at European level, such as ARROW and the Orphan Works Database, established by the European Union Intellectual Property Office, in accordance with the Orphan Works Directive. The Member States also contributed to the Framework for an EU-Wide Audiovisual Orphan Works Registry (FORWARD project), which ran from 2013 to 2017. The current crisis made the need to improve online access more compelling, as acknowledged by stakeholders. In particular, stakeholders pointed to various obstacles when accessing and reusing cultural heritage online, such as a lack of sufficient content, insufficient quality, the copyright and reuse status. They further highlighted that obstacles to encouraging co-creation, co-curation and other forms of interaction persist. Likewise, collaboration with the archives is not sufficiently facilitated to promote an innovative work. Moreover, for both metadata and content data there is insufficient multilingual content and more needs to be done to promote this. With respect to Europeana, the respondents to the consultation generally acknowledged that Europeana had a positive impact on the sector, with spillover effects in education (see below) and the creative industries. As mentioned in Section 3, the indicative quantitative targets were achieved earlier than planned. However, while images and text were the main categories, there was only a marginal contribution of 3D digitised material despite the generally acknowledged significant potential and value of 3D digitisation technologies. Various initiatives promoted the use of 3D technologies within the sector. For instance, the 3D Content in Europeana Taskforce was set up to identify ways to enhance the accessibility of 3D collections and Member States collaborated on several joint EU projects. As mentioned above, additional efforts to promote quality content, including multilingual content, need to be made. In terms of communication plans to raise awareness of Europeana among the general public and notably in schools, the Recommendation appears to have been less effective, as only seven Member States reported a national communication plan for this action. Since the Recommendation’s adoption, various collaborative projects have taken place, such as the ones between the Europeana Foundation and the European Schoolnet, bringing digital cultural heritage to classrooms across Europe and providing further opportunities for digital teaching and learning. However, feedback from consultations highlighted the need to further enhance teachers’ awareness of Europeana. Similarly, Europeana’s potential through CC0 licensing should be better emphasised through collaboration with creative industries. Using Europeana, and more particularly the Europeana Data Model, the Europeana Publishing Framework and the International Rights Statements to develop a standardisation approach in the cultural heritage sector has been important not only in the EU, but also at international level, where our models and best practices have been replicated. Box 1. Several of Europeana’s key achievements, i.e. common standards and solutions, such as the Europeana Data Model, the International Rights Statements and the Europeana Publishing Framework, have been taken up widely in the cultural heritage sector in Europe and internationally (e.g. in the Digital Public Library of America). However, as mentioned above, unlocking funding from ministries remains an important challenge overall. Figure 2 below indicates how the member States’ financial contribution to Europeana increased in 2011 (when the Recommendation was adopted), to sharply decrease afterwards. Figure 2. Ministry funding for Europeana 2010-2018 (in EUR) Source: Evaluation study (Europeana annual reports and accounts, 2010-2018) The provisions on digital preservation encourage Member States to have appropriate and up-to-date measures for the digitised cultural material, while promoting coordination at EU level. As shown by the evaluation study, in 2014 only 26 % of CHIs had created a written digital preservation strategy that was endorsed by the organisation’s management, increasing to 42% by 2017. Stakeholders confirmed the Recommendation’s contributions to increase digital preservation. Various initiatives have been taken since the Recommendation’s adoption, including the establishment of digital repositories and competence centres, the adoption of new software and technologies for digital preservation (e.g. the use of a repository of e-books on demand in Austria and the Nestor catalogue in Germany). As shown in Section 3 above, there has been limited progress on multiple copying and migration, with half of the Member States having explicit legislative provisions to allow such activities by public institutions. Although there has been clear progress in coordinating approaches and preventing significant disparities across the EU in legal deposit arrangements, there is considerable scope for improvement. Most of the Member States currently have measures in place to allow preservation by using techniques collecting material from the internet, such as web-harvesting techniques, although for some this precedes the Recommendation’s adoption (e.g. in Denmark specific provisions were laid down through the 2005 Legal Deposit Law and the process is carried out by the Royal Danish Library, using the NetArchive web-harvesting platform). Nonetheless, participants in the November 2019 DCHE workshop highlighted the need to strengthen and update the provisions on digital long term preservation, in order to meet the needs of the sector, in line with technology developments.

In order to be fully effective, the Recommendation should also clearly cover all the forms of cultural heritage not addressed at all or only minimally addressed, e.g. born digital, intangible (e.g. stories, dances or traditions, performances) or immovable. This is supported by the feedback from the online public consultation, where the majority of respondents (65%) were in favour of both broadening the scope of the Recommendation to include cultural heritage currently not addressed, and including the digital transformation of cultural heritage institutions. Moreover, digitisation of intangible cultural heritage has been considered very important by more than half of the overall responses (54%), while 37% considered it important. There is a convergence of views across the stakeholder groups, from 58% of NGOs considering it ‘very important’ to 47% of public authority respondents, but also a convergence of views across the sectors (e.g. tourism full agreement). Based on the feedback from the consultation activities, and as illustrated above, the objectives of the Recommendation can be considered to have been achieved overall. The evaluation study pointed to the following factors that have either contributed to or hindered the achievement of the objectives of the Recommendation: a) Positive factors: The Member States’ expert group, the DCHE, has been a useful forum for the Recommendation’s implementation and for the exchange of best practices among the Member States. It has also been acting as a governance body for the Europeana Digital Service Infrastructure, providing strategic orientations for its future and contributing to the sustainability of Europeana. More recently, the DCHE was also instrumental in getting 25 Member States to sign the 2019 Declaration of Cooperation on advancing the digitisation . This has also shown this group’s effectiveness for forward-looking initiatives. Tools have been designed to support CHIs to become more confident about using digital to deliver on their missions, i.e. tools to design, organise and carry out digital cultural heritage activities. The following examples can be mentioned: the Europeana Impact Playbook, a step-by-step approach to help CHIs find inspiration, practical resources and networks to more efficiently achieve organisational goals; the Culture24 project promoted training and upskilling programmes to drive up digital capabilities across the heritage sector, or the GIFT BOX, a package of open-source tools and recommendations that help CHIs to make digital experiences richer for their visitors and to enhance engagement. The Europeana Publishing Guide helps data partners to share collections in Europeana, bringing the existing information to one place and outlining the metadata and content requirements for publishing in Europeana. Standardisation: Europeana has been particularly successful in the provision of standards regarding the publication and sharing of data that have been taken up in the cultural heritage industry across the EU and internationally, for example by the Digital Public Library of America (DPLA). This is, in particular, the case of the Europeana Data Model (EDM), the Europeana Publishing Framework (EPF) and the RightsStatements.org international initiative. The case study done for the evaluation of the Europeana initiative in 2018 highlighted that the Europeana Publishing Framework had a positive impact on the overall quality of data, although there was still a lot of scope for further improvement. Technological advancements promoting open-source software and shared data have improved the online access of cultural heritage content and promoted collaboration between CHIs, with researchers, private or creative sectors, etc. Examples include specific factors like the FLOSS inventory for digital cultural heritage and digital humanities, listing all the Free Libre Open Source Software (FLOSS) relevant to the cultural heritage sector, or developments in cloud storage (helping CHIs maintain archives through digital preservation) and in linked data practices (promoting an open culture of online accessibility and reuse in the digital cultural heritage sector). Opportunities, initiatives and collaboration provided by EU funding for the digital cultural heritage community (e.g. under the Framework Programmes for Research and Technological Development, Connecting Europe Facility or Horizon 2020) were a key factor that helped to achieve the Recommendation’s objectives. b) Negative factors: Lack of adoption of technical common standards and approaches at regional level for digitised content impeded the achievement of the Recommendation’s intended objectives (due, in particular, to legal constraints or incompatible file formats). The need for more interoperability of standards (including semantic) appears to still be a challenge for the sector, as stated in the workshops. Similarly, insufficient user-friendly multilingual services must be addressed. Participants in the workshops held by the Commission highlighted the need to enhance the promotion and application of the FAIR principles, i.e. digital data should be findable, accessible, interoperable and reusable. Insufficient advanced digital skills and literacy in the cultural heritage sector are some of the key issues highlighted in the consultations. The digital shift has created new needs for the custodians of cultural heritage and new opportunities to be seized to support new forms of visitor experience and participation. Research for the evaluation study emphasised that insufficient advanced digital skills and literacy have an impact on the availability of the digitised content, on the efficiency of the process and on the possibilities for innovation. As underlined also in the feedback from the online consultation, the need for a constant update of knowledge and skills in line with state-of-the-art technologies concerns all the digitisation related workflow and processes, from digitisation itself to access and preservation. Up-skilling staff with relevant and comprehensive digital learning mechanisms leads to enhanced digital skills and knowledge embedded in their work. Insufficient funding for CHIs was also singled out as a major challenge by the respondents in the consultation activities, in particular the difficulty to shift from project funding to stable sources of funding. CHIs, in particular the small ones, struggle to secure a sustainable source of funding, and this has an impact on digitised content and innovation. The wide ranging impacts of the COVID-19 crisis underlined once more the need for the CHIs to get support for their activities, in numerous cases also to ensure their long-term survival (see Section below). 5.2. Efficiency This section assesses the costs and benefits associated with the Recommendation’s implementation, more precisely the extent to which the Recommendation achieved its objectives at a proportionate cost. Key point on the baseline: üVery often a national budget was not allocated for digitisation activities. üFor digitisation, CHIs mainly used their own resources (or national and EU funds for projects). Key findings on the current situation:

üThe absence of available data to quantify the costs of digitisation and preservation triggered by the Recommendation’s implementation was a significant challenge, making it difficult to draw conclusions and make comparisons across Member States. üEven if a more detailed analysis of the proportionality costs - benefits is difficult to carry out, stakeholders underlined that the benefits outweighed the costs. üMost of the costs were borne by the Member States (with an uneven budget allocation across them) and by CHIs; üThe Recommendation provided an opportunity for more discussions, exchanges and collaborations for the cultural heritage sector, resulting in more visibility and better prioritisation and thus triggering more funding. üThe benefits were enjoyed by the whole of society: CHIs have benefitted from attracting and leveraging funding, collaborative networks and wider resources, expanding their audience; governments have benefitted from growth in cooperation and information sharing; creative sectors, teachers, students, the public in general have benefitted from a wider access to digitised cultural heritage resources. üOverall, more than 900 cultural institutions across Europe have benefitted from EU support to make their collections accessible through Europeana, and to carry out targeted digitisation, while 50 033 909 cultural heritage items are digitally available through Europeana. Respondents to the online public consultation gave positive feedback on the Recommendation’s efficiency, with around 70% of them either agreeing or strongly agreeing that the benefits of implementing the Recommendation are significant and justify the costs. Most of the costs have been borne by a) Member States (governments) and b) CHIs, while benefits have been enjoyed by everybody/society as a whole (e.g. users, creative sectors, teachers, students). The main limitation in this section is the absence of available data to quantify the costs of digitisation triggered by the Recommendation’s implementation. The reports do not always provide relevant or consistent data, making it difficult to establish causal links and to draw conclusions. Additional evidence confirms this and highlights the fact that national support for digitisation emerges from a combination of programmes (not necessarily directly linked to the digitisation of collections) and it is thus challenging to track financial allocations for the digitisation of cultural heritage (along with the scope of the heritage field, the cultural policy structure, the coordinating structure). Some of the respondents in the targeted consultation pointed to the challenges in quantifying the costs or barriers arising from the Recommendation. a) Costs for the Member States: Most of the evidence in this section relies on the budget allocated to digitisation (with budgets only as an indirect proxy for costs), together with qualitative analysis, i.e. information and views collected from the consultation activities organised by the Commission or in the context of the evaluation study. Moreover, as highlighted by the evaluation study, financial efforts for digitisation were already underway before the Recommendation’s adoption and they may have continued, but to what extent is uncertain. However, during the consultation activities, Member States and stakeholders generally acknowledged that the Recommendation provided an opportunity for the cultural heritage sector to engage in more discussions, exchanges and collaborations, resulting in more visibility and better prioritisation and thus triggering more funding. The main resources allocated by the Member States to digitisation are presented in Figure 3. Figure 3. Budgets (in EUR) and sources for digitisation of cultural material (2011-2017) Country 2011-2013 budget Sources 2013-2015 budget Sources 2015-2017 budget Sources Belgium 2 800 000 N 200 000 N 450 000 N Bulgaria 586 778 EU Croatia 370 000 384 196 N Germany 16 500 000 N 19 300 000 N 599 000 N 800 000 N 100 000 N 2 500 000 N Greece 170 000 000 EU 100 000 000 EU 4 500 000 EU 1 000 000 EU Estonia 5 000 000 EU 110 000 N Spain 65 000 N 900 000 N 400 000 N 585 000 N 173,103 N 92 800 N 70 000 N France 2 000 000 N 9 631 299 N Latvia 11 900 000 EU 2 100 000 N Netherlands 121,600 000 N 5 000 000 N Romania 11 000 000 N Poland 28 000 000 N 28 369 465 N 101 000 000 EU Slovakia 194 000 000 EU 141 000 000 EU 141 000 000 EU Slovenia 8 000 000 EU Source: Evaluation study. While the figures presented above suggest that reporting of the allocated budgets is uneven, it is, however, clear that some Member States have been allocating substantial funds for digitisation activities. Larger Member States have been providing larger amounts and smaller Member States (in particular in southern and eastern Europe) have had smaller budgets and have been relying mainly on EU funds. For instance, in Germany, Saxony allocated EUR 2.5 million and Baden-Württemberg EUR 0.6 million. However, the evaluation study highlights the limited and inconsistent data provided by the Member States over the period covering 2011-2017 (i.e. only 14 Member States), without a clear separation between national resources and EU funds. This make it difficult to draw conclusions and make comparisons across Member States. b) Costs for CHIs

Most of the costs are borne by Member States (governments) and CHIs, the latter being responsible for digitisation. The costs of the CHIs have been used as a proxy for the costs stemming from the Recommendation, although it is not possible to establish causal links and a direct impact. Feedback to from the online public consultation pointed to the fact that the cultural sector has been working with declining budgets despite an increase in technology related costs and with the pressure for more digitisation (and online access). As shown by the evaluation study, based on the ENUMERATE surveys, the average total costs per institution doubled between 2012 and 2017, from EUR 176 000 to EUR 446 000 (Figure 4). Total costs grew depending on the type and size of the CHI. However, it is important to note that increasing the volume of digitised cultural heritage material resulted in higher costs. Figure 4. Average total costs per institution for digital collections (in EUR thousand) Source: Evaluation study Evidence shows that labour costs increased over time, with more full-time equivalents (FTE) in digital collection activities, from 5.5 FTE in 2012 to 11 FTE in 2017 (05). In terms of paid employment, the average FTE grew from 3.5 FTE to 7 FTE in the period. The increase in digitisation activities required, in addition to more full-time paid employees, volunteers, with an increase from two in 2012 to four in 2017. The growth in employment can be considered a positive effect of the Recommendation’s implementation. Figure 5. Average FTE in digital collection activities per CHI, 2012-2017 Source: Evaluation study Moreover, as shown by the evaluation study, between 2013 and 2017, the proportion of costs remained the same, with incidental costs (e.g. for the selection of material, the initial creation or acquisition of a digital collection) representing 53 % and structural costs (e.g. ongoing maintenance, enhancement and preservation of a digital collection) 47 % of the total costs. In some Member States, costs are also borne by private sources. In the Netherlands, private funding is increasing, although some sources are more for projects involving the digitisation and reuse of archival materials for publications, for example. In Estonia there are only a few of these private funds (e.g. the Soros Fund before closure). Benefits of the Recommendation’s implementation: Despite the difficulty in quantifying such benefits, during the consultation activities stakeholders highlighted the benefits of the Recommendation for the whole of society. a) Benefits for the Member States As mentioned above, participants in the DCHE meeting in November 2019 noted that the Recommendation was important in promoting the sector and enhancing cooperation among the Member States, while fostering better organisation and coordination internally. This led to better prioritisation, allocation and a more efficient use of financial resources. Moreover, raising public awareness of and through digital cultural heritage assets (e.g. see Section 2.2.2. on Europeana) has led to spill-over effects in other domains, e.g. tourism. b) Benefits for the cultural heritage sector As acknowledged during the consultation activities, the Recommendation’s implementation largely benefitted the sector by exchanging information and more generally know-how, by promoting and sharing tools, networks and connections and by facilitating access to resources and digital activities that might not have been carried out without the Recommendation. It created additional opportunities to gain new perspectives from working with peers and experts in digital cultural heritage. This has been particularly relevant for smaller CHIs, whose access to funding and networks is more limited. In this regard, the Europeana Network Association has been very successful in bringing together heritage professionals, tech providers, educators, researchers and copyright specialists. Its six communities (Europeana Tech, Communicators, Copyright, Education, Impact and Research) allow numerous CHIs to network, learn and exchange best practices. Collaboration between the tech industry and the cultural heritage sector required joint forces, creating efficiency gains, stimulating innovation and enhancing the economic potential of Europe’s cultural heritage sector. Likewise, developing synergies with other public institutions stimulated progress on the use of new advanced technologies. For instance, the Tuscany Region in Italy and the University of Florence established the New Media for Cultural Heritage Competence Centre (NEMECH), connecting research centres and institutions, promoting the transfer of research know-how and providing digital technologies tools and applications for cultural heritage.

As mentioned above, increased visibility of the sector led to increased support for cultural heritage and access to more funding and infrastructure. Overall, more than 900 cultural institutions across Europe benefitted from EU support to make their collections accessible through Europeana, and to carry out targeted digitisation. This raised awareness among the cultural heritage sector on the importance of having a digital presence, has advanced the digital experience of cultural heritage institutions and mobilised cultural institutions to make their material accessible online. More than 3 700 institutions have contributed to Europeana and 50 033 909 European cultural heritage items are digitally accessible through it. Also before the Recommendation, several countries did not have digital strategies, the Recommendation helped stimulate the adoption of such strategies and thus led to more funding becoming available for digital projects. Moreover, the greater number of discussions and exchanges of information have enabled cultural heritage custodians to better understand their needs and challenges in a rapidly evolving technological context. For instance, capacity building and particularly the upskilling of staff (not mentioned in the Recommendation) became an important element to consider in order to get prepared to efficiently and effectively use the opportunities provided by the digital transformation. As pointed out by the evaluation study, although there is no direct or quantifiable evidence of the Recommendation’s impact on other sectors (e.g. tourism, education and research), the responses during the consultation activities highlighted the Recommendation’s contribution to the rest of the economy within and across Member States and the synergies with other projects and initiatives. Box 2. The EU-funded APEx project (Archives Portal Europe network of eXcellence) ran from March 2012 to September 2015, funded under ICT Policy Support programme, as part of the Competitiveness and Innovation framework programme (CIP). The APEx project involved 33 partners from archival and research organisations from 30 countries, and hundreds of participating and content providing archives. APEx's overall goal was to provide easy access via the Archives Portal Europe (APE) to as much archival content of European institutions as possible and to channel all digitised and digital archival material to Europeana. APEx was by far one of the biggest archival cooperation projects in Europe. The main result from the project was the Archives Portal Europe, including substantially improved stability and capacity and usability, and significantly enriched content; this gave users online access to a great amount of European archival data (and metadata). c) Benefits for the general public and society at large As explained in the sections above, some of the benefits of the Recommendation, as mentioned during the consultation activities, include wider access to previously restricted content and democratisation of culture and knowledge, grater efforts to innovate and develop new technologies and services with spillover effects ino other sectors (e.g. tourism, research, education). Moreover, the Recommendation stimulated policies, strategies and projects at EU, national and local levels for the benefit of the whole of society; it encouraged co-creation and co-curation, while keeping people and communities connected. Historiana is a good example of an initiative promoting virtual learning for educational purposes; its collaboration with Europeana aimed to create new ways for history educators to create, share and use e-learning activities with content from the Europeana collections website. Responses to the consultations indicated that improving digital skills and digital literacy leads to further efficiency gains, as mentioned above. The Recommendation should take into account how digital content is produced and used by cultural heritage professionals, creative sectors, research, education, users, etc. Up-skilling staff with relevant and comprehensive digital learning abilities leads to enhanced digital skills and knowledge embedded in their work. Doing sowould lower costs and maximise resources. Similarly, increased collaboration and common initiatives, sharing of know-how and networks would lead to a more efficient use of resources and the fulfilment of organisational missions. Figure 6. Comparison of benefits and costs Stakeholder Benefits Costs General public, creative sector, teachers, students and researchers Access to content +++ Learning ++ Co-creation and co-curation + CHIs and professionals Funding and collaboration networks ++ Incidental and structural costs +++ Digital skills + Labour costs ++ Audience expansion and public mission + Member States Co-operation and information sharing ++ Digitisation funding + Legend: the '+' indicates the Recommendation’s impact, ranging from small (+), medium (++), and large (+++). Source: Evaluation study During the consultation activities respondents highlighted that digital transformation is not an option anymore, it is a reality, it is a requirement of the current technological development of society and CHIs must embrace and support it. This may translate (initially) into additional costs (e.g. in infrastructure, in digital skills and literacy), but also benefits (e.g. more agile processes, improved knowledge and visibility, reaching wider audiences). Results can be achieved with fewer resources only once the required changes have been implemented and the processes are optimised and working. Increased collaboration between private and public, as well as between cultural heritage professionals, will lead to efficiency gains, while greater synergies with other sectors will benefit the whole of society. Overall, it was acknowledged that implementing the right digital tools and acquiring the necessary skillsets can help cultural heritage professionals to optimise processes and resources, while building confident CHIs and engaged users. However, in a context where the COVID-19 crisis strongly hit the sector, threatening the long-term financial sustainability of many CHIs, funding becomes even more important in a context where governments will most likely make additional budgetary cuts. 5. 3. Relevance The main question addressed in this section is whether and to what extent the Recommendation has been and is still relevant to the needs of the stakeholders. This section looks at how well the Recommendation is adapted to future technological and social advances.

Key points on the baseline: üDespite the efforts and progress of the Member States and CHIs to digitise, preserve and provide online access to digitised cultural heritage content, further steps were needed. üThe need for more digitisation, online access and preservation, along with the need for more harmonised efforts across the EU to unlock Europe’s economic and cultural potential made the case for the Recommendation. Key findings on the current situation: üStakeholders generally acknowledged the Recommendation’s relevance in responding to the needs identified at the time of its adoption; üEvidence pointed not only to the Recommendation’s relevance, but also to the need to strengthen it to adapt it to the current needs and challenges. üSocio-economic changes, technological advancements, legislative changes create a different context for CHIs that must be taken into account. The Recommendation needs to better reflect the huge potential of advanced digital technologies (e.g.3D digitisation) and emerging technologies (including immersive technologies such as VR or AR and AI), bringing unprecedented opportunities for digitisation, online access and preservation. üThe need for advanced digital literacy and skills is one of the key issues highlighted in all the consultation activities. ü It is important to make full use of cultural heritage data for the benefit of the sector and to enhance interoperability and capacity building. üIntangible and born-digital cultural heritage need to be taken into account in any future revision of the Recommendation. The feedback gathered from the consultation activities revealed a general acknowledgement of the relevance of the Recommendation in responding to the needs identified at the time of its adoption. As mentioned above, the Recommendation sought to address the need to a) unlock Europe’s economic and cultural potential, b) better harmonise national approaches across Member States to digitisation and the preservation of cultural heritage, as well as c) increase digitisation, online access and preservation to avoid losing our cultural heritage assets and their memory (due to floods, fires, mass tourism, etc.). At the time of the Recommendation’s adoption, the context for digitisation efforts was changing: more people were consuming culture online, CHIs needed to change tools and organisational culture and the creative sectors, which are a driver for innovation and economic growth, were flourishing. Managing the ‘digital shift’ required a strategic approach defining how creativity could be better promoted and how access to and the use of digitised cultural resources could be facilitated, while preserving them for the next generations. Stakeholders and Member States representatives acknowledged the Recommendation’s contribution in encouraging public authorities to prioritise and increase support for digitisation and provide more funding, while increasing online access and preservation. Using digital for creation and culture led to unlocking the potential for broader societal, cohesive and economic benefits from sectors such as sustainable tourism, education and creative sectors. The impact was thus on the whole of society: on CHIs (e.g. able to reach out to broader audiences and engage with new users), creative sectors (e.g. stimulation of creativity and content), and teachers, students, the public (e.g. wider access to cultural material), etc. As mentioned in Section 5.1, participants in the DCHE workshop of November 2019 highlighted the importance of the Recommendation in helping CHIs to participate in the Europeana network and its activities, thus encouraging the sharing of knowledge and best practices. Through Europeana, the Recommendation has also played a significant role in the digital development of the cultural heritage sector in Europe by strengthening cooperation and standardising cross-border activities. Moreover, the Recommendation’s relevance through Europeana and more particularly through the synergies created within the sector of education, i.e. to provide further opportunities for digital teaching and learning (see Section 5.1.) has been confirmed by the responses to the 2017 Eurobarometer on cultural heritage, where a large majority of respondents (88%) agree that Europe's cultural heritage should be taught in schools. Box 3. In March 2017 the Europeana4Education initiative was launched to help inspire learners and enrich educational resources with Europeana content. Europeana created a dedicated education area on Europeana Pro, allowing visitors to explore partnerships, browse case studies of educational applications, resources and platforms featuring Europeana content, and get practical help from the collection of resources. Today, Europeana Classroom provides a wealth of innovative learning for educators and learners with a selection of educational resources using digital culture. Partnerships with EuroClio, the European Association of History Educators, for example, have produced useful case studies that provide insights into the use of online collections for and by educators. As indicated in Section 2 above, evidence points to the high number of objects that were not digitised before the Recommendation and to the key role it had in promoting and supporting digitisation and online access to digitised content. This further helped to avoid the complete loss of the memory of cultural heritage assets due to natural or human derived factors.

While the outcome of the consultation activities revealed a general acknowledgement of Recommendation’s the relevance in responding to the relevant needs and achieving its objectives, the consultations also revealed a strong need to strengthen the Recommendation, to make it better reflect the current technological, economic and societal state of play. More particularly, 95% of the respondents to the online public consultation agreed that the Recommendation should be updated to reflect today’s needs in the area and increase the potential of Europe’s cultural heritage. Since the Recommendation’s adoption, there have been significant and interlinked changes in the context in which digitisation, online access and digital preservation take place: socio-economic changes leading to different and increased user needs, technological changes (e.g. AI, machine learning), legislative changes (e.g. on copyright). These changes resulted in new opportunities and challenges for the CHIs and guided their choices and actions. The majority of the responses from the DCHE representatives in the targeted consultation pointed to the fast paced changes in the digital environment that the sector needs to cope with and urged the Commission to revise the Recommendation to keep it relevant for the future. As emphasised by the evaluation study, better access to and a greater use of the internet, together with increased consumption of online goods and services, higher cultural employment and government expenditure on recreation, religion and culture are all examples of what has helped to create a new environment for both providers and users of Europe’s digitised cultural heritage. The development of these factors is shown in Figure 6. Figure 6. User characteristics in relation to internet use and the culture sector, EU-28 average, 2011 - 2017/19 2011 Most recent data available Change Internet use Households – level of internet access 73 % 90 % (2019) + Individuals – internet use 73 % 88 % (2019) + Internet use: purchasing cultural goods and services Films and music 17 % 17 % (2018) + Books, magazines and newspapers 21 % 22 % (2018) + Tickets for events 21 % 27 % (2018) + Films and music delivered or upgraded online 8 % 14 % (2017) + Cultural engagement Cultural employment (absolute value) 7.9 million 8.7 million (2018) + Cultural employment (% total employment) 3.6 % 3.8 % + Persons working as creative and performing artists, authors, journalists and linguists 1.8 million 2 million (2018) + Government expenditure Recreation, religion and culture 2.3 % 2.3 % (2017) = Cultural services 1.0 % 1.0 % (2017) = Source: Evaluation study The 2017 Eurobarometer reveals that over half of the respondents (55%) have used the internet in the last 12 months for at least one of a range of cultural heritage purposes, and a large part of them (31%) for the accessibility, facilities and main features of a museum, historical monument or traditional event in preparation for a visit or their holidays. In addition, almost one quarter (23%) used the internet for buying or booking services for events or activities (e.g. tickets, guided tours, etc.) while 21% used it for viewing cultural heritage-related content, such as the description of a work of art or historical monument during a visit, historical information about a traditional event they attend; 11% of the respondents have created or shared cultural heritage-related content, such as a picture or a video of a work of art or historical monument, etc. This shows a high interest of Europeans in getting a wider virtual access to cultural heritage and confirms the Recommendation’s objectives and relevance. The responses to the online public consultation highlighted the sector’s efforts to increase online access during confinement, through more digital learning resources (e.g. recording lectures and support material to be streamed); more digital activities (such as extending online access to curated audio-visual assets, virtual tours of exhibitions), increased social media presence and online collaboration. The COVID-19 pandemic has acted like a catalyst for increased online activities and services, with the majority of respondents stating that they envision additional changes by their organisations in the future (e.g. better procedures, better quality of the digital content, more frequent use of collaboration platforms especially for teaching and research activities). However, if on the one hand the crisis accelerated the digital transformation of the heritage sector, on the other it also deepened the current inequalities between large and small CHIs (i.e. the latter not having the same resources to share their collections and materials online, missing out on digital opportunities). The development of digital technologies such as AI, computer vision, deep learning, machine learning, cloud computing, Big Data, VR and AR has brought unprecedented opportunities for digitisation, online access and preservation. Digital technologies can empower and encourage people to participate in culture in a more active and creative way. In particular, AI allows manifold uses of diversified and rich data across Member States and CHIs. For instance, it has a great potential to extract knowledge from different data sets of cultural artefacts held by CHIs across Member States. AI can allow users to navigate through extremely vast amounts of cultural and creative content, while reducing the efforts and resources required for such effort by CHIs. AI can also boost automated annotation of digitised cultural heritage, or can help to reconstruct lost cultural heritage information or to extract new knowledge from cultural heritage data. As underlined in the feedback from the public consultation, VR/AR technologies can enhance the user experience, while providing great potential for use in education and research. 3D technologies are now mature enough to help in preservation and restoration or in providing virtual access to cultural heritage where it is difficult or impossible to get (e.g. under water). These technologies provide numerous opportunities for CHIs to reach wider audiences, with more immersive experiences, for both onsite and online visitors. The need to focus more on higher quality and more particularly on 3D emerged from both the consultation activities, and from the reports of the Member States. For instance, a large majority of overall respondents to the public online consultation consider it very valuable or valuable to create digital twins of cultural heritage buildings, monuments and sites such as Notre Dame, ahead of any damage by fire or other disasters and more than half of the respondents (53%) agreed that 3D technologies are very valuable for creating high-quality 3D models of museum objects (in addition to 38% agreeing it is valuable). There is a clear need to strike the right balance between quantity and quality, moving away from the current approach focused mostly on quantity. In this respect, stakeholders argued that since the Recommendation’s adoption, the emphasis has been on increasing the volume of digitised cultural assets and ensuring a high quality for all of them (including metadata) and this remains a challenge. Therefore, in order to remain relevant, the Recommendation should address the role advanced technologies can play in the field (e.g. strengthening the ways CHIs could deliver higher quality content, supporting new forms of visitor experience and engagement), while fostering inclusivity, equality, diversity. It is important to note that a large majority of the respondents to the public consultation (and also across the sectors) agreed that, based on the lessons learned from the COVID-19 crisis, digital technologies would be very important in making their organisation more resilient in the future while helping to increase the number of people accessing cultural heritage through the internet. Participants in the workshop on digital transformation held in Brussels (2020) indicated the need to properly address the issue of making full use of cultural heritage data for the benefit of the sector. Cultural heritage data are currently not sufficiently collected, used and reused within this sector or across other sectors (e.g. education, research) and the Recommendation would need to properly reflect this. Social media and the existing web data could, for example, be an important source of data and more efforts should be placed on this. In terms of repositories or other elements of data infrastructure, maximising their impact through a wide scale initiative is important. The respondents to the online public consultation (96%), considered common data spaces for sharing cultural heritage data and building high-value data sets to be valuable instruments in the field. This view was particularly shared by stakeholders such as academic or research institutions, public institutions, the public or sectors such as the technology /ICT, cultural, education and research sectors. Likewise, respondents to the online consultation considered repository infrastructure for digitised cultural heritage content (95%), cloud infrastructure (90%), supercomputing capabilities for large-scale cultural heritage simulations (76%) key to cultural heritage. While the Recommendation has contributed to more interoperability in the sector by supporting the development of a framework of interoperability and increasing awareness of common frameworks, there is still work to be done for instance on semantic interoperability or interoperability of standards. Ensuring appropriate means for preservation implies not only sustaining single digital objects, but also creating the right context for them to be understood in the future. Overcoming technology obsolescence is a key element in avoiding a complete or partial loss of digital objects and is still a complex task. Users have become more demanding with regard to the products and services they consume/use, wanting to participate, enjoy and learn more. They challenge the CHIs more and require a stronger link with them, i.e. to maximise and diversify the possibilities to digitally interact with cultural content. According to stakeholders, in order to embrace digital transformation, a key factor that professionals of the sector need to address is capacity building. In this respect, providing staff with advanced digital skills and literacy remains a big challenge for many of them. In particular, half of the respondents to the public online consultation confirmed that their organisation had struggled with the advanced digital skills needed to manage their online presence during the pandemic. Respondents working in the education and research sectors were likely to be the most affected, followed by respondents from the culture sector. Additional evidence collected during the pandemic confirms that many CHIs in rural areas did not have the necessary digital skills to carry out their activities during the lockdown, leading to a significant void in culture on offer. In addition to the large variety of CHIs and the many different tasks that staff need to carry out, knowledge and skills should be constantly updated and kept in line with technological developments. Feedback from stakeholders, particularly from the January 2020 workshop pointed to the importance of having a common standard and classification of skillsets for cultural heritage professionals (e.g. for senior and middle management, as well as expert levels), depending on the specific needs and domains (e.g. legal, technical, life-cycle management, communication). Other stakeholders pointed out for example that researchers and practitioners do not have the knowledge and ability to master digital tools and devices to deal with the accelerated development of ICT. Ensuring the necessary (specific) skills and capacities of the professionals of the sector is a priority for the creation, management and development of digitised cultural heritage, with spillover effects ino other sectors. Feedback from the online consultation highlights what stakeholders consider to be the core set of digital skills that professionals from the cultural heritage sector should acquire: data and metadata management and analytics, and copyright and licensing, followed by digitisation (2D and/or 3D). Furthermore, findings from the DCHE workshop in November 2019 highlighted the importance of a higher competence level for staff that would drive up capabilities across the heritage sector. At a more general level, in the consultation on the strategic objectives of the Digital Europe programme, held between 25July and 25October 2019, 74% of the respondents indicated that the EU should support the Member States in improving the advanced digital skills necessary to operate the up-to-date digital infrastructures. Likewise, a large number of responses considered EU support for culture very important. New forms of cultural heritage have emerged, and their collection and preservation remain a challenge, as indicated by stakeholders during the consultation activities (e.g. for born-digital heritage, meaning it has been created digitally, without an analogue equivalent or cultural content created outside CHIs, such as on social media or the gaming industry). Due to a rapidly evolving technology, with hardware and software becoming quickly obsolete, born-digital cultural heritage needs to be properly collected, managed and preserved to be accessible and usable in the long run.

While in the short term, most of the CHIs have been focusing on the immediate challenges raised by the COVID-19 pandemic, new models and ways of working will emerge in the longer term. CHIs will learn from the pandemic, by creating innovative approaches for digital literacy and skills, processes, content and communication, through a new organisational culture and mindset. 5. 4. Coherence This section assesses how the various actions have worked together to achieve the objectives. It looks particularly at the extent to which the Recommendation is aligned and consistent with other EU policies, with Europeana- related policy/initiatives and with cultural heritage policies in the Member States. Key points on the baseline: üThe Recommendation has created synergies with the European Agenda for Culture (2007), helping to shape the policy framework for the cultural sector. üIt has built on the work done through Europeana (2008). üAt the time of its adoption, the Recommendation was an important action for one of the key areas tackled by the Digital Agenda for Europe, a flagship initiative of the Europe 2020 strategy (2010). Key findings on the current situation: üOverall, the Recommendation is aligned with several other policy interventions in different fields, such as culture or copyright. In particular, evidence suggests that the Recommendation is consistent with the New European Agenda for Culture, the Work Plan for Culture covering the period 2019-2022, the European Framework for Action on Cultural Heritage, the Digital Agenda for Europe, the Digital Single Market strategy, the Directive on copyright and related rights in the Digital Single Market and the Directive on open data and the reuse of public sector information. üHowever, a future revision of the Recommendation should aim to ensure better alignment with these more recent initiatives such as the new digital strategy ‘Shaping Europe’s Digital Future’ to create a Europe fit for the digital age and with ’A European strategy for data’. A majority of the respondents to the online public consultation (around 59%) considered that the Recommendation had been coherent with other EU policies and initiatives. The Recommendation is consistent with several other EU policy interventions in various fields, such as culture and copyright, or is aiming, more generally, to unlock Europe's innovative capabilities, and to exploit the economic and societal benefits of a digital society. In the field of culture, the Recommendation has been sharing the same objectives and efforts with more specific actions in the field, to promote culture and its key role in Europe’s society and economy. For instance, the Recommendation has created synergies with the European Agenda for Culture, adopted in 2007, helping to shape the policy framework for the cultural sector. By encouraging the Member States to take actions for greater digitisation, online accessibility and preservation, the Recommendation has contributed to the objectives of the European Agenda for Culture, namely: ·promotion of cultural diversity and intercultural dialogue; ·promotion of culture as a catalyst for creativity in the framework of the Lisbon Strategy for growth and jobs; ·promotion of culture as a vital element in the Union's international relations. The Recommendation was an important action for the Europe 2020 strategy of March 2010, and in particular for its flagship initiative ’A Digital Agenda for Europe’, aiming to deliver sustainable economic and social benefits through a Digital Single Market. Providing active support for the digitisa