COMMISSION STAFF WORKING DOCUMENT on the evaluation of Europeana Accompanying the document Report from the Commission to the European Parliament and the Council on the evaluation of Europeana and the way forward
Date: 2018-09-06
Institutions: Directorate-General for Communications Networks, Content and Technology, European Commission
Subjects: EU programme, Internet, cultural heritage, cultural transmission, digital archiving, digital technology, digitisation, heritage protection, virtual library

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 6.9.2018  SWD(2018) 398 final  COMMISSION STAFF WORKING DOCUMENT on the evaluation of Europeana Accompanying the document Report from the Commission to the European Parliament and the Council  on the evaluation of Europeana and the way forward {COM(2018) 612 final}  
 
 
Table of contents 

1. 

2. 

3. 

INTRODUCTION ................................................................................................................................ 3 

BACKGROUND TO EUROPEANA ................................................................................................... 3 

STATE OF PLAY ................................................................................................................................. 4 

4.  METHOD ............................................................................................................................................. 6 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ............................................ 6 

6. 

CONCLUSIONS ................................................................................................................................. 13 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 15 

ANNEX 2: STAKEHOLDER CONSULTATION (SYNOPSIS REPORT) ................................................ 17 

ANNEX 3: EVALUATION QUESTIONS .................................................................................................. 23 

1 

 
 
Glossary 

Term or acronym 

Meaning or definition 

Connecting Europe Facility (CEF) 

Digital Service Infrastructure (DSI) 

Europeana (or Europeana initiative) 

The EU funding instrument that supports the development of high 
performing, sustainable and efficiently interconnected trans-
European networks in the fields of transport, energy and digital 
services (telecom). 

Digital infrastructure providing trans-European interoperable 
services of common interest for citizens, businesses and/or public 
authorities, funded under the Connecting Europe Facility (CEF).  

Europeana is an initiative, built through a series of EU-funded 
projects, that has created a platform for unique access to Europe’s 
digitised cultural heritage resources, developed common standards 
and solutions to achieve data interoperability and accessibility, and 
nurtured cross-border visibility and use of cultural resources. 

Europeana operator (or Europeana 
DSI operator) 

The consortium operating the Europeana Digital Service 
Infrastructure, under grant or procurement by the Commission.  

Europeana network (or ecosystem) 

The network of cultural heritage institutions, domain and national 
aggregators and professionals contributing to Europeana. 

Europeana Network Association 
(ENA) 

Association of professionals working in the digital cultural 
heritage and related sectors. 

Content 

Metadata 

The digital reproductions available through the Europeana DSI. 

Set of text data that describes the cultural objects. 

Europeana Data Model (EDM) 

The data model developed and used by Europeana.  

Europeana Publishing Framework 
(EPF) 

Framework setting out four scenarios (tiers) for sharing content 
with Europeana. 

RightsStatements.org 

Set of standardised rights statements communicating the copyright 
and re-use status of digital objects available online. 

Application Programming Interface 
(API) 

An API enables two software programs to communicate with each 
other. Europeana APIs enable developers to access and use the 
data from Europeana’s platform in their programs and 
applications. 

2 

 
 
 
1. 

INTRODUCTION 

Purpose and scope 

On  31 May  2016,  the  Education,  Youth,  Culture  and  Sports  (EYCS)  Council  adopted 
conclusions on the role of Europeana for the digital access, visibility and use of European 
cultural heritage1. The Council Conclusions confirmed Member States’ continued support 
for Europeana, recognised Europeana’s relevance both from a cultural as well as a digital 
innovation perspective, identified challenges to be addressed, and invited Member States, 
the Commission and the Europeana operator to undertake action. 

In these Council Conclusions, the Commission was invited: 

‘to  present  to  the  Council  an  independent  evaluation  of  Europeana  and  give  clear 
orientations  for  the  mid-  and  long-term  development  of  Europeana  by  assessing 
alternatives at the EU level for the future scope, sustainable funding and governance of 
Europeana, including a possibility to transform or integrate Europeana into a European 
legal entity, whilst taking account of the dual nature of Europeana as both a cultural and 
digital innovation project;’ 

In response to the Council’s request, the Commission carried out an  evaluation of 
Europeana based on the five mandatory criteria of the Commission’s Better Regulation 
Guidelines2.  Europeana  was  assessed  in  terms  of  its  (i) relevance,  (ii) effectiveness, 
(iii) efficiency,  (iv) coherence,  and  (v) EU  added  value.  The  scope  of  the  evaluation 
covered  both  the  underlying  concept  and  value  of  Europeana  as  a  European 
cultural  and  digital  innovation  project,  as  well  as  an  in-depth  review  of  the 
adequacy  and  viability  of  the  services  provided  by  Europeana  as  a Digital  Service 
Infrastructure (DSI) under the Connecting Europe Facility (CEF)3. 

The  evaluation  results  provided  input  to  an  analysis  of  future  options  and  to  the 
formulation of orientations for its mid- and long-term development, as requested by the 
Council.   

The  outcomes  of  the  evaluation  will  be  transmitted  to  the  Council  and  Parliament  and 
will further inform future activities of the Commission. 

2.  BACKGROUND TO EUROPEANA 

Europeana  was  launched  in  2008  as  ‘The  European  Digital  Library’  to  make  Europe's 
cultural  heritage.  Its  creation  was  initiated  by  a  letter  from  six  Heads  of  State  and 
Government  to  the  Commission  in  2005.  Financial  and  political  support  has  continued 
ever  since4  for  coordinating  and  integrating  EU  Member  States’  efforts  to  digitise  and 

1 The role of Europeana for the digital access, visibility and use of European cultural heritage — Council 
conclusions (31 May 2016) 
2 European Commission Better regulation: guidelines and toolbox 
3 Evaluation Roadmap 
4 The timeline of the digitisation and online accessibility of cultural heritage outlines all relevant initiatives 
and policy documents since 2005. 

3 

 
 
 
                                                            
make  cultural  material  accessible  online,  including  standardisation,  interoperability  and 
cooperation among cultural heritage institutions across Europe. Europeana contributes to 
the implementation of a number of policy areas. These include the Digital Single Market, 
the  Commission  Recommendation  of  27 October  2011  on  the  digitisation  and  online 
accessibility  of  cultural  material  and  digital  preservation  (2011/711/EU)5  and  related 
Council Conclusions (EYC Council, May 2012)6, calling on Member States to get more 
material online and ensure the long-term preservation of digital material. 

Today, Europeana is Europe’s digital platform for cultural heritage, funded as a well-
established  Digital  Service  Infrastructure  (DSI)  under  the  Connecting  Europe  Facility 
(CEF), to: 

-  bring Europe’s digitised cultural heritage material online and to promote its cross-

border visibility and use across Europe; 

-  offer solutions for the trans-European interoperability and accessibility of digital 

- 

resources of European heritage; 
further improve the conditions for re-use and facilitate take-up of digital cultural 
content  and  metadata  in  other  sectors,  e.g.  research,  education,  tourism  or  the 
creative industries; and 

-  provide  a  multilingual,  user-friendly  access  point  to  Europe’s  rich  and  diverse 

heritage. 

Europeana offers services creating value for different user groups: 

-  data  partners 
professionals; 

(contributing  cultural 

institutions)  and  cultural  heritage 

-  end-users (anyone with an interest in culture); 
- 

re-users (education, research and creative sector professionals).  

Europeana promotes standardisation, best practice and capacity building among cultural 
heritage institutions and professionals in the area of cultural and digital innovation. 

Europeana contributes to preserving and fostering European culture in the digital 
age  and  involves  all  EU  Member  States.  It  supports  the  objectives  of  the  European 
Year  of  Cultural  Heritage  2018  to  encourage  the  sharing  and  appreciation  of  Europe’s 
cultural heritage, to raise awareness of common history and values, and to foster a sense 
of  belonging  to  a  common  European  space.  Europeana  is  the  only  digital  cultural 
heritage partner of the European Year of Cultural Heritage 2018. 

3.  STATE OF PLAY 

Europeana currently provides access to over 51 million items from the collections of over 
3 700 libraries, archives, museums, galleries and audio-visual collections across Europe, 
through  its  main  portal  ‘Europeana  Collections’  (europeana.eu)  and  through  the 
platform’s  application  programming  interfaces  (APIs).  The  material  that  is  accessible 
through  Europeana  includes  books,  manuscripts,  audio  and  video  recordings,  national 

5 Commission Recommendation of 27 October 2011 on the digitisation and online accessibility of cultural 
material and digital preservation (OJ L 283. 29.10.2011, p. 39) 
6 Council Conclusions of 10 May 2012 on the digitisation and online accessibility of cultural material and 
digital preservation (OJ C 169, 15.6.2012, p. 5). 

4 

 
 
                                                            
archives,  documents  illustrating  Europe's  history,  manuscripts  of  major  writers  and 
composers, as well as digital copies of artworks. 

Following  the  Council  Conclusions  of  2016,  Europeana  refined  its  Strategy  2020  and 
focused on three priorities: 

-  Making  it  easy  and  rewarding  for  cultural  heritage  institutions  to  share  high 

quality content. 

-  Scaling with partners to reach target markets and audiences.  
-  Engaging people on Europeana’s websites and via participatory campaigns. 

A consortium led by the Europeana Foundation is operating the current Europeana DSI 
core  service,  i.e.  the  central  platform  and  services.  Europeana  Foundation  cooperates 
with  (i) a  number  of  domain  and  thematic  aggregators  that  support  the  aggregation  of 
material from cultural heritage institutions, (ii) partners supporting distribution and re-use 
of  Europeana  data,  and  (iii)  expert  partners  supporting  different  technical  aspects.  A 
number  of  national  aggregators  support  Europeana  in  improving  the  aggregation 
infrastructure and data quality. The Europeana Foundation has  led a series of consortia 
operating  the  Europeana  initiative  since  its  launch  in  2008.  A  number  of  user-oriented 
projects, supporting thematic collections, tools and re-use, have been funded as generic 
services since 2016. 

From  2008  to  August  2017,  Europeana  was  funded  through  grants.  Its  scope  and  main 
objectives  were  set  in  the  annual  work  programmes  of  the  funding  instruments. 
Throughout this period, the implementation strategy of Europeana was largely set out by 
Europeana Foundation, as leader of the winning consortia. 

Since  grants  require  co-financing,  the  winning  consortia  had  to  identify  additional 
sources of revenue to cover all the costs necessary to run Europeana. Voluntary Member 
States'  contributions  have  always  been  a  challenge  as  Member  States  considered  that 
their contribution was through digitisation of content and contribution of that content to 
Europeana. 

The  Commission  has  made  provisions  for  stable  continuous  support  to  the  Europeana 
platform.  Following  the  Council  Conclusions  of  2016,  the  Commission  changed,  as  of 
September 2017, the funding model for Europeana’s core platform and services to one of 
procurement  while  continuing  to  fund  generic  services  projects  and  activities  through 
grants. 

The  Europeana  Foundation  Governing  Board  of  stakeholders  and  experts  from  the 
cultural heritage community meets five times per year to set the implementation strategy 
and review plans on goals and objectives. 

The Europeana Network Association is a community of cultural heritage, creative, and 
technology  professionals,  who  exchange  and  promote  best  practices  and  cross-border 
cooperation.  Through  its  Members  Council,  it  provides  input  on  Europeana’s  strategy 
and  activities.  Its  Management  Board  (six  members  elected  by  the  Members  Council) 
represents the Europeana Network Association on the Governing Board of the Europeana 
Foundation. 

Member  States  influence  the  direction  of  Europeana  through  the  CEF  Telecom 
Committee,  the  Commission’s  Expert  Group  on  ‘Digital  Cultural  Heritage  and 
Europeana’, and participating on the Europeana Foundation Governing Board. 

5 

 
4.  METHOD 

The evaluation of Europeana was conducted through the following process: 

  An  independent  comprehensive  study  was  carried  out  from  July  2017  to  May 
2018 for gathering and analysing data and for assessing the evaluation questions. 
The  tasks  of  the  study  included  refining  the  evaluation  questions  and 
methodology,  data  gathering  (including  desk  research,  case  studies,  interviews, 
benchmarking,  technical  usability  evaluation  and  analysis  of  the  public  online 
consultation), and assessing the evaluation questions. 

  A  panel  of  five  independent  experts,  from  different  fields  of  relevant  expertise, 
were  appointed  by  the  Commission7.  The  experts  validated  the  evaluation 
questions and provided feedback, comments and a final assessment. 

  The  Commission  also  carried  out  a  12-week  public  online  consultation,  open 
between  17 October  2017  and  14 January  2018,  in  all  EU  official  languages.  A 
targeted questionnaire, with more technical questions, addressed to professionals 
and  organisations  that  had  experience  with  Europeana’s  infrastructure  and 
professional  network  was  included  in  the  public  consultation.  The  public 
consultation results have provided significant input to the evaluation. 

  The  entire  process  was  followed  by  an  Interservice  Steering  Group  with 
representatives  from  different  Commission  DG  to  ensure  the  quality  and 
impartiality of the process. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

The  main  findings  of  the  evaluation,  based  on  the  five  evaluation  criteria,  and 
corresponding evaluation questions, can be summarised by the following. 

5.1.  RELEVANCE 

The evaluation has  highlighted that,  with over  51 million  items,  Europeana provides 
access  to  one  of  the  largest  digital  collections  of  cultural  material  in  the  world. 
Europeana further stands out by offering material in a large number of languages (37 
languages), as well as in the variety of its content (text, image, 3D, audio-visual, sound) 
from  a  broad  network  of  data  providers  from  different  domains  (e.g.  archives, 
libraries,  museums).  The  benchmark  study  also  shows  that  Europeana  targets  a  wider 
scope of audiences (cultural heritage institutions and professionals; research; education; 
creative  industries;  culture  professionals;  individuals)  than  other  cultural  heritage 
platforms analysed. However, this makes it more difficult for Europeana to satisfy every 
user group’s needs. 

7 The panel of experts were also tasked with analysing scenarios and options for the future development of 
Europeana, based on the evaluation outcomes. See Annex I. 

6 

 
 
 
 
 
                                                            
Europeana’s relevance to EU policies and priorities for the online accessibility and 
dissemination  of  European  cultural  heritage  is  rated  high.  The  concept  and  core 
values of Europeana fully support EU policy8 by bringing the common cultural heritage 
to the fore, enabling European cooperation, and expressing an inclusive narrative about 
the diversity and interconnectivity of culture in Europe. Furthermore, Europeana supports 
the  EU  Digital  Single  Market  policies  related  to  online  accessibility  and  digital 
preservation  of  cultural  heritage  by  improving  the  availability  and  re-use  of  digital 
cultural heritage across sectors and across national borders. 

The  respondents  to  the  public  online  consultation  agree  that  the  portal  ‘Europeana 
Collections’9 is valuable and useful in various aspects: Europeana Collections advances 
cultural  knowledge  and  learning  (82 %);  Thematic  collections  and  exhibitions  allow 
exploration  of  specialised  content  (85 %);  Trustworthiness  of  content  sources  (82 %); 
Availability  of  materials  in  different  languages  (79 %).  Two  thirds  of  respondents 
indicated  that  Europeana  was  important  for  finding  and  exploring  European  cultural 
heritage. 

According to the findings of the case studies and the public consultation, Europeana’s 
services are relevant to cultural heritage institutions for the online accessibility and 
dissemination  of  European  cultural  heritage.  Specifically  for  small  cultural  heritage 
institutions, the possibility for them to share data in Europeana is perceived as beneficial 
because it increases the visibility and recognition of their collections. 

At  least  two  thirds  of  the  respondents  that  provided  data  to  Europeana  indicated  that 
Europeana  generated  value  to  their  organisation  via  audience  reach,  added  value  to 
content, and facilitation of partnerships with other cultural institutions. 

Europeana’s  relevance  as  a  facilitator  of  professional  exchange  of  cultural  and 
technological  expertise  is  recognised  in  the  findings  of  the  case  studies  and  the  public 
consultation  analysis.  Europeana’s  past  and  current  activities  to  find  common 
solutions and to set up frameworks (such as the Europeana Publishing Framework 
and  the  international  standardisation  work  under  the  RightsStatements.org 
initiative) are seen as very important for the digital cultural heritage sector. Cultural 
institutions  benefit  from  best  practices  as  far  as  common  standards  are  concerned  and 
from joining and learning from a pan-European network. 

Two  thirds  of  the  respondents  that  indicated  that  they  have  been  members  of  the 
Europeana Network Association (ENA) stated that the exchanges through the ENA have 
been  beneficial  to  them,  and  that  the  ENA  complements  other  digital  cultural  heritage 
networks. 

While  Europeana  has  exceeded  its  objectives  in  terms  of  quantity  of  items  the 
relevance and quality of its content and metadata is an issue. Although at least two 
thirds  of  the  public  consultation’s  respondents  stated  they  were  satisfied  with  various 
aspects of Europeana’s content, most responses to the open question elaborating on this 
answer provided constructive criticism. These mainly pointed towards the need for more 
high  quality  material  and  for  material  from  currently  underrepresented  European 

8 Treaty on the Functioning of the European Union (TFEU), Article 167. 

9 https://www.europeana.eu/ 

7 

 
                                                            
countries,  certain  institutions  or  of  specific  themes.  They  also  stated  that  the  quality  of 
the content was unsatisfactory, particularly the resolution and colours of images. Rights 
statements were also considered to be restrictive in some cases and therefore to preclude 
re-use. 

In  terms  of  metadata,  the  lack  of  translation,  its  varying  quality  (poor  or  incorrect 
metadata), and inconsistencies (often due to different metadata formats used by different 
institutions)  have  a  negative  impact  on  the  service’s  overall  popularity  and  affect  the 
findability of an object when searched for on Europeana. A number of negative responses 
in  the  public  consultation  concerned  the  lack  of  relevance  of  search  results.  High-level 
quality  data  is  a  key  parameter  for  Europeana,  and  cultural  heritage  institutions  play  a 
central  role  in  this  regard,  being  ultimately  responsible  for  the  quality  of  the  provided 
content  and  metadata.  The  recent  update  of  Europeana’s  strategy,  focusing  on 
strengthening  the  quality  of  data  and  making  the  existing  (and  incoming)  content  on 
Europeana more interesting and valuable to end-users and re-users, is highly relevant. 

5.2.  EFFECTIVENESS 

Europeana reaches a significant number of user profiles, whose interests in the platform 
are quite diverse. Public consultation results indicate that most respondents are from the 
education, culture and research sectors. 

Most targeted groups use the portal less than once per month (33 %) or a few times per 
month (27 %). The highest proportion of respondents indicated that they re-used content 
from Europeana in research projects or as teaching materials (e.g. in presentations). 

The  diversity  of  users  and  their  interests  in  Europeana  provide  a  challenge  in 
maintaining high quality services for all targeted groups. 

The evaluation study shows that the current performance indicators of Europeana are not 
sufficient  for  monitoring  progress.  Moreover,  the  statistics  and  analytics  covering 
previous years are not readily accessible nor presented in a systematic manner. Statistics 
must be published on a regular basis in order to provide the most up-to-date data to data 
partners and other stakeholders. It is recommended that further data gathering on reach 
and  scope  is  considered,  firstly,  to  identify  where  further  investigation  can  be  focused 
and,  secondly,  as  a  way  of  segmenting  Europeana’s  performance  with  the  different 
audiences. 

In terms of technical infrastructure, Europeana is on a par with the other platforms 
in the benchmark, offering the interfaces and functionality expected from a hub for the 
exchange of cultural information. According to the benchmark analysis, Europeana is on 
a  par  with  the  other  examined  platforms  in  aspects  such  as  interoperability  with  data 
providers  and  re-users;  aggregation  of  metadata  and  content,  procedures  for  metadata 
import  and  mapping,  data  management  and  multilingual  indexing.  However,  there  is 
room  for  significantly  improving  the  aggregation  infrastructure  and  workflow,  and  the 
functionality of the Europeana portal. 

One of Europeana’s main achievements was to be able to gather thousands of individuals 
and  organisations  (in  particular  cultural  heritage  institutions)  from  very  different 
backgrounds and enable them cooperate in a dynamic and expanding Europeana Network 
to share metadata about their content in a single place. However, the findings highlight 
the need to (i) make the aggregation process easier for data providers, (ii) reduce the 

8 

 
time  from  data  submission  to  publication,  as  well  as  (iii)  provide  better  tools  and 
technical  support  for  the  data  ingestion  process.  Case  study  interviewees  stated  that 
these  actions  are  very  important  for  engaging  and  motivating  the  cultural  heritage 
institutions, which in turn are crucial for increasing the overall data quality. 

57 %  of  the  respondents  of  the  public  consultation  were  satisfied  with  the  ease  of 
searching and filtering results, while only 48 % were satisfied with the ease of navigating 
through multilingual material, and 53 % with the relevance and accuracy of results. 

Those who answered the question on the extent to which they were satisfied with finding 
what they searched for were asked to elaborate on their answer. The negative responses 
pointed  mainly  to  the  lack  of  features  on  the  website  and  the  lack  of  relevance  of  the 
search results. Specifically, respondents pointed to a lack of sophisticated filtering as a 
method of refining a search and a need for more advanced searching possibilities for 
the  portal  and  the  platform’s  APIs.  Respondents  also  reported  search  results  to  be 
inconsistent,  containing  broken  links,  lacking  thumbnails,  and  containing  duplicates  or 
redundant hits. 

Another issue is the portal’s level of multilingualism: Only 46 % of respondents to the 
public consultation indicated satisfaction with multilingualism of metadata. The majority 
(59 %) of the respondents indicated that they use English for their searches. Respondents 
pointed specifically to the lack of translation of metadata and to inconsistencies in search 
results when searching in different languages. 

The well-curated collections on the portal have significantly higher satisfaction and 
engagement rates. Additionally, user engagement campaigns like Europeana 1914-1918 
(collection  of  family  stories  and  memorabilia  related  to  WW1  provided  by  the  public) 
proved  successful  and  innovative  in  reaching  and  engaging  a  wider  audience  in  the 
locations where they took place. 

API  services  were  seen  as  very  interesting  by  the  developers  that  took  part  in  the 
usability  evaluation.  Nevertheless  the  search  engine  did  not  always  provide  relevant 
results  or  the  results  contained  broken  or  blank  links.  The  API  documentation  was 
sometimes found to be incomplete or inaccurate. 

82 % of data partners agree or strongly agree that Europeana's promotion of open culture 
and open cultural data has advanced and facilitated their work.    

One  of  Europeana’s  greatest  achievements  lies  in  providing  standards  and 
frameworks regarding the publication and sharing of data that have been taken up 
in the cultural heritage sector across the EU and internationally, for example by the 
Digital  Public  Library  of  America  (DPLA).  This  is,  in  particular,  the  case  of  the 
Europeana  Data  Model  (EDM),  the  Europeana  Publishing  Framework  (EPF)  and  the 
the  Europeana 
RightsStatements.org 
Publishing Framework shows that its use has brought progress on the overall quality of 
data, although there is still a lot of scope for improvement since the biggest part of the 
material (around 64 %) is currently in Tier 1 (lowest tier out of four). 

initiative.  The  case  study  on 

international 

On communication activities, 21 % of the respondents wanted to see more efforts made 
to  raise  awareness  about  Europeana.  Respondents  mainly  emphasised  that  awareness-
raising activities should, on the one hand target the general public, for instance, through 
greater presence on social media and improved search-engine optimisation. On the other 

9 

 
hand,  many  respondents  also  highlighted  students  and  researchers  as  key  groups  to  be 
targeted  for  awareness-raising  activities,  as  these  are  main  re-users  of  Europeana’s 
content. 

At  least  50 %  of  the  professionals  who  had  re-used  material  from  Europeana  indicated 
being unaware of Europeana’s activities in encouraging re-use in education, research and 
the  creative  sectors,  except  for  Europeana’s  hackathons.  Only  a  small  proportion  (less 
than  7 %)  of  respondents  indicated  that  they  were  aware  of  and  engaged  in  such 
activities. 

In  terms  of  adapting  to  a  fast  changing  technological  and  user-driven  environment, 
Europeana’s technical infrastructure is progressively improving, through the continuous 
re-design and improvement of its platform and portal. From a technical point of view, the 
data  model,  the  publishing  framework  and  the  interoperability  procedures  offer  a  basis 
for further developments. The main challenge is the speed and effectiveness with which 
the  Europeana  organisation  and  governance  structure  will  be  able  to  react  to  new 
requirements.  Technical  review  reports  covering  the  period  under  evaluation  show 
examples of where plans and investments started but did not conclude, or only did so in a 
minor way. Europeana’s ability to adapt  to change is also influenced  by the fact  that  it 
relies on data provided by participating content providers. The challenges these providers 
have  in  transforming  their  collections  in  a  rapidly  changing  technological  environment 
are inherited by Europeana. The case study on small and medium sized institutions shows 
that Europeana adds value in upgrading knowledge, skills and tools at cultural heritage 
institutions level. 

5.3.  EFFICIENCY 

Europeana’s infrastructure faces a number of challenges and constraints regarding 
efficiency. 

A  major  challenge  is  the  current  aggregation  infrastructure.  It  is  based  on  domain 
and national aggregation, which has the benefit of dealing only with a couple of hundred 
direct partners to bring in data from 3 700 institutions. The drawbacks are the complex 
structure  and  content  submission  process  and  the  amount  of  time  needed  from  data 
submission to publication, often leading to a lack of motivation for data providers. 

The  public  consultation  identified  several  reasons  for  why  data  is  not  provided  to 
Europeana: 31 %  of  the  respondents  described a  lack  of  time  or  resources,  and  25  %  a 
lack of information on how to provide data as major barriers to contribution. 

Of  the  respondents  that  provided  data  to  Europeana,  only  around  half  indicated 
satisfaction with the various aspects of Europeana’s aggregation structure. This question 
was elaborated upon in an open answer, with 76 % providing constructive criticism and 
suggestions  for  improvement.  11 %  of  the  responses  stated  they  were  dissatisfied  with 
the inability to update content and/or metadata after it had been provided to Europeana, 
and 9 % stated they were dissatisfied with the communication with Europeana regarding 
support  and  feedback  on  the  data  provision  process.  This  was  particularly  the  case  for 
some  cultural  heritage  organisations  that  had  contributed  to  Europeana  through  EU-
funded projects that have now ended, often leading to a ‘publish and forget’ practice. 

10 

 
 
84 % of respondents that provided data to Europeana indicated that in future they were 
likely  to  continue  supplying  data  via  an  aggregator,  and  58 %  that  they  were  likely  to 
provide data directly to Europeana. 

Case  study  analysis  indicates  that  domain  aggregators  (including  best  practice 
networks  such  as  AthenaPlus,  EU  Screen  or  Photoconsortium)  and  national 
aggregators  help  cultural  institutions  to  share  their  data  and  guarantee  a  correct 
provision of data to Europeana. This assistance is considered as particularly beneficial 
for  smaller  cultural  institutions.  The  case  study  also  highlighted  that  the  national 
aggregators are better equipped to survive and link to Europeana even though not every 
country has one. 

Despite the above-mentioned limitations, Europeana notably managed to build a wide 
ecosystem  or  network  of  aggregators  and  cultural  heritage  institutions  that  play  a 
key role in providing input at both national and European level. Europeana’s governance 
involves  a  large  part  of  the  initiative’s  stakeholders  and  is  well  established  to  handle 
changing  challenges  while  remaining  relatively  efficient.  Some  limitations  relate  to  the 
need  for  Europeana  to  be  more  inclusive  and  to  ensure  a  greater  participation  of 
aggregators  and  cultural  heritage  institutions,  as  well  as  to  clarify  the  role  and 
responsibilities of all players involved in the organisation. 

The funding model for Europeana’s core service platform based on grants did not 
provide a sufficiently stable basis for funding Europeana, since it required co-funding 
from  other  sources.  Despite  the  efforts  of  the  Commission,  the  Member  States  and  the 
Europeana Foundation,  the  grant  model  proved  to  be  unsuitable  for  sustaining  the  core 
service.  In  the  2016  Council  Conclusions,  Member  States  reached  an  agreement  to 
continue  their  contributions  to  Europeana  until  the  Commission  progressively  converts 
its support scheme from a grant model to a procurement model that could cover all core 
service costs. This shift took place in September 2017. 

Since  it  started  in  2007,  consecutive  EU  funding  programmes  (eContentplus  (2005-
2008),  Competitiveness  and  Innovation  Programme    (2007-2013),  CEF  (2014-2020)) 
have  contributed  to  the  development  and  deployment  of  the  Europeana  initiative. The 
funding  has  supported  both  the  central  coordination  and  the  European  integration  of 
national efforts to make digital cultural material accessible through Europeana, including 
standardisation, interoperability and targeted digitisation. 

Funding  of  approximately  EUR  10  million  a  year  under  the  current  CEF  programme 
focuses on maintaining and further developing Europeana’s core platform service, while 
also supporting, since 2016, user-oriented projects as generic services. The EU funding 
has  been  entirely  used  to  support  the  development  and  deployment  of  the  Europeana 
infrastructure.    Europeana  is  not  an  initiative  associated  with  regulatory  costs,  savings, 
burden reduction in Member States or similar, but serves as a pan-European set of digital 
and innovation services for cultural heritage that would otherwise not exist. As such, the 
assessment  of  Europeana  is  based  on  the  benefits  and  impact  that  the  initiative  has 
achieved. 

Quantitative data on the socio-economic benefits and impact generated by Europeana is 
not available. However, the Europeana initiative has had a high leverage factor: whereas 
the  Commission  supported  directly  around  900  organisations  to  make  their  collections 
accessible  through  Europeana,  and  to  carry  out  targeted  digitisation,  the  Europeana 
initiative  has  raised  awareness  among  the  cultural  heritage  sector  on  the  importance  of 

11 

 
having  a  digital  presence,  has  advanced  cultural  heritage  institutions’  digital  expertise 
and has mobilised cultural institutions to make their material accessible online. This has 
led more than 3 700 institutions to contribute to Europeana (leverage factor of 4:1) and 
over  51  million  European  cultural  heritage  items  can  be  accessed  digitally  through 
Europeana. 

5.4.  COHERENCE 

The evaluation confirms the coherence of Europeana as regards wider EU policy, in 
line with the Gothenburg communication of November 2017 on Strengthening European 
Identity  through  Education  and  Culture10.  With  regard  to  strengthening  the  sense  of 
European  identity  and  awareness  of  cultural  heritage,  the  Gothenburg  communication 
designates  the  2018  European  Year  of  Cultural  Heritage  as  the  occasion  to  show  how 
digital tools can broaden access to cultural material and opportunities, and to shed light 
on initiatives such as Europeana, which fosters access to cultural heritage material held 
by libraries, archives and museums through digital means. 

Europeana  is  overall  coherent  with  the  national  strategies  for  the  digitisation  and 
online accessibility of cultural heritage material of the Member States. Europeana, to a 
certain extent, addresses challenges that the EU Member States face in digitising cultural 
heritage  and  transforming  access  to  culture.  It  complements  national  initiatives  with 
similar objectives and is the only pan-European platform of its kind. 

90 %  of  the  respondents  indicated  there  is  some  degree  of  complementarity  between 
Europeana  and  national  initiatives  (such  as  Gallica,  Hispana  or  the  German  Digital 
Library).  These  respondents  highlighted  in  particular  Europeana’s  role  as  the  only 
organisation  that  provides  access  to  cultural  material  across  borders  in  Europe,  while 
61 % also indicated there was some degree of overlap with these. 

Europeana’s work on, for instance on standards, tools and open access,  is seen as 
useful  across  all  sectors  influenced  by  digital  transformations,  even  if  not  all 
stakeholders agree to the specific solutions promoted. 

Europeana  maintains  overall  internal  coherence,  its  activities  and  services  being 
consistent  with  each  other,  even  though  evaluation  findings  state  that  further  effort 
should  go  into  improving  certain  aspects,  such  as  internal  communications,  having  a 
sufficient number of technical staff, focusing on enabling core technology and narrowing 
the scope of activities. 

5.5.  EU ADDED VALUE 

According to the evaluation study, Europeana has played a significant role in the digital 
development of the cultural heritage sector in Europe. Case study analysis indicates that 
strengthening  cooperation  and 
Europeana’s  efforts  have  been  key 
standardisation  activities  across  borders  in  line  with  the  overarching  goal  of 
increasing  the  use  of  standards  in  digitising  and  sharing  digitised  cultural  material 

to 

10 Strengthening European Identity through Education and Culture, COM(2017) 673 final, 14.11.2017 

12 

 
  
 
                                                            
throughout  Europe.  The  Europeana  Network  of  aggregators,  cultural  heritage 
institutions  and  professionals  is  seen  as  an  important  component  of  cross-border 
and pan-European collaboration, leading to interoperability and standardisation in the 
digital cultural heritage sector. 

35 %  of  respondents  stated  that  there  had  been  developments  at  European-level,  in 
response  to  the  question  on  developments  that  would  not  have  happened  without 
Europeana,  referring  mostly  to  strengthened  cooperation  within  the  cultural  heritage 
sector.  27 %  stated  that  there  had  been  national-level  initiatives  referring  mostly  to 
Europeana’s contribution to standardisation of metadata and creation of national portals 
in their countries.  

Overall, projects developed under the Europeana initiative enabled new developments 
to emerge at both national and European level. 

At least two thirds of respondents indicated that they believe Europeana brings value to 
the  EU,  particularly  in  digitally  bringing  together  and  providing  access  to  cultural 
heritage items from across Europe (85 %), standardising the format of object descriptions 
(70 %), and facilitating a European network of digital heritage professionals (70 %). 

The main topic raised by the positive comments was that Europeana has underpinned 
a  sense  of  a  shared  history  and  identity  among  European  citizens  (for  example, 
‘Unifying European history, reminding us that we have a long and rich shared past not 
without problems [ …] Europeana’s importance is obviously more than a platform with 
pretty images’). 

6.  CONCLUSIONS 

The  evaluation  has  highlighted  an  overall  appreciation  of  the  European  added  value  of 
Europeana  by  cultural  heritage  institutions  across  the  EU.  Europeana  is  a  unique 
initiative and the only pan-European platform of its kind. It is the only aggregator in the 
world that brings together the largest digital collection from across 28 Member States. It 
has  also  helped  inspire  a  sense  of  shared  history  and  identity  among  European  citizens 
through  its  unique  collections,  such  as  the  popular  1914-1918  collection  recalling  the 
First World War. At the same time, it has brought together European cultural institutions, 
enabling  them  to  collaborate  and  share  their  material  with  European  citizens.  Not  only 
has  this  led  to  sharing  of  best  practices  on  common  standards,  but  also  it  has  helped 
nurture  a  European  network  of  data  partners,  aggregators  and  professionals  in  various 
fields who have stimulated the capacity building and exchange of expertise. Some of the 
major  achievements  are  the  progress  towards  a  de  facto  standardisation  in  the  cultural 
heritage sector though the Europeana Data Model, the Europeana Publishing Framework 
and the International Rights Statements. 

Europeana also complements national strategies of Member States for the digitisation and 
online  accessibility  of  cultural  heritage,  and  reflects  EU  policy,  such  as  the  overall 
objectives  of  the  Gothenburg  communication  of  November  2017  on  Strengthening 
European Identity through Education and Culture. 

However, the evaluation results also highlight that several transformations are needed in 
a number of areas. For example, there are significant challenges for Europeana stemming 
from its focus on a large number of diverse activities appealing to a wide range of user 

13 

 
 
groups.  Narrowing  down  the  focus  of  efforts  and  concentrating  resources  to  reach 
maximum quality and impact of the offered services would be necessary. 

Better  quality  of  services  remains  a  persistent  demand  from  many  respondents  of  the 
open  public  consultation,  be  they  cultural  heritage  institutions  or  individual  users  of 
Europeana.  Quality  related  issues  are  interlinked  between  quality  of  content  and 
metadata,  the  findability  of  material,  and  multilingualism  functions.  The  findability  of 
content  would  be  improved  if  more  metadata  was  translated  or  was  available  in  more 
than one language. 

Strengthening  the  technical  platform  core  is  a  condition  sine  qua  non  for  Europeana  to 
remain a reliable cultural content and service provider. The evaluation further highlights 
the need to strengthen the aggregation infrastructure for Europeana to be able to service 
and  support  the  cultural  heritage  institutions  in  their  actions  towards  capacity  building 
and digital transformation.  

14 

 
 
 
 
ANNEX 1: PROCEDURAL INFORMATION 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

This  Evaluation  Staff  Working  Document  was  prepared  by  Directorate  G  ‘Data’ /  Unit 
G.2 ‘Interactive Technologies, Digital for Culture and Education’ of Directorate-General 
‘Communications Networks, Content and Technology’ (DG CNECT). 

The  Decide  planning  reference  of  the  initiative  ‘Evaluation  of  Europeana’  is 
PLAN/2016/55. 

2.  ORGANISATION AND TIMING 

  A panel of five independent experts from different fields of relevant expertise were 
hired  to  assist  the  Commission  in  evaluating  Europeana  and  assessing  possible 
options for its future development. 
Names of the experts (alphabetically): 
-  Prof  Dieter  Fellner,  Director,  Fraunhofer  Institute  for  Computer  Graphics 

Research IGD, Germany; 

-  Ms  Laure  Kaltenbach,  Creative  Entrepreneur,  co-founder  of  CreativeTech 

Agency and ex-Managing Director, Forum d’Avignon, France; 

-  Dr Jarred McGinnis, Independent Consultant in Semantics, United Kingdom; 
-  Dr Piotr Rypson, Deputy Director of the National Museum in Warsaw, Poland; 
-  Ms  Gianna  Tsakou,  Project  Manager/Senior  Analyst,  SingularLogic  S.A., 

Greece. 

  The  Commission  procured  an  independent  study11  which  was  carried  out  by  a 
consortium led by the consultancy firm CARSA12 between July 2017 and May 2018. 
The tasks of the study included: 

o  refining the evaluation questions and methodology; 
o  gathering and analysing evidence data; and 
o  assessing the evaluation questions and drafting the report. 

  An 

(ISG)  with 

Interservice  Steering  Group 

from  DG 
Communications  Networks,  Content  and  Technology,  DG  Education,  Youth, 
Sport  and  Culture,  DG  Research  and  Innovation,  DG  Informatics,  and  the 
Secretariat-General  has  followed  the  entire  process.  The  ISG  approved  the 
evaluation roadmap and stakeholder consultation strategy, the terms of reference and 
the deliverables of the external study, as well as the online questionnaire of the public 
consultation. 

representatives 

11 EU Bookshop: Evaluation of Europeana and orientations for its future development, following adoption 

of Council Conclusions by EYCS Council on 31/05/2016 (SMART no 2016/0100) 

12  Consortium  composed  of  CARSA  (lead  partner),  Ramboll  Management  Consulting,  Agilis  and  SQS 

(subcontractor). 

15 

 
 
 
 
                                                            
3.  EVIDENCE, SOURCES AND QUALITY 

The  Commission,  supported  by  the  external  contractor,  gathered  qualitative  and 
quantitative evidence from the following sources. 

  A  public  online  consultation  on  Europeana  ran  between  17 October  2017  and 
14 January  2018,  in  all  EU  official  languages.  The  consultation  generated  1 221 
responses,  of  which  883  were  from  individuals  and  338  were  on  behalf  of  an 
organisation.  Among  the  individuals,  66 %  indicated  that  their  main  interest  in 
Europeana was personal use and 61 % professional use. 

  A  targeted  questionnaire,  with  more  technical  questions,  addressed  to  professionals 
infrastructure  and 

that  had  experience  with  Europeana’s 

and  organisations 
professional network was included in the public consultation. 

  Literature review 
  Four case studies, including two interviews per case study: 

o  The value of the Europeana Network; 
o  Europeana Publishing Framework — activities and impact;  
o  Re-use of Europeana in education; and  
o  Benefits for cultural institutions that share data in Europeana. 

  Benchmarking between Europeana and seven other organisations that have developed 

digital cultural heritage projects: 

in EU Member States: the British Library, Finna.fi, BnF, Rijksmuseum; 

o 
o  International organisations: The World Digital Library, The Digital Library of 

America (DPLA), Google Arts & Culture. 
  Usability evaluation of the platform at three levels: 

technical evaluation; 

o 
o  evaluation with user and developer groups; 
o  automated  and  quantified  evaluation  of  the  user’s  perception  in  terms  of 

availability and response time. 

  The  national  progress  reports  on  digitisation,  online  accessibility  and  digital 
preservation  of  cultural  material  submitted  by  Member  States  for  the  period  2015-
2017.  A  section  of  the  reports  concerns  the  Member  States’  contributions  to  the 
Europeana initiative. 

  CEF sustainability study and CEF mid-term evaluation. 

The  panel  of  experts  conducted  additional  informal  interviews  with  the  Executive 
Director and the Deputy Director of the Europeana Foundation (operator), representatives 
of  the  Europeana  Foundation  Governing  Board  and  representatives  of  the  Europeana 
Network Association Management Board. 

16 

 
 
 
 
ANNEX 2: STAKEHOLDER CONSULTATION (SYNOPSIS REPORT) 

1. 

INTRODUCTION 

Following  the  Council  Conclusions  of  31  May  201613,  the  European  Commission 
launched an independent evaluation of Europeana (www.europeana.eu). The scope of the 
evaluation  covered  both  the  underlying  concept  and  value  of  Europeana  as  a  European 
cultural  and  digital  innovation  project  and  an  in-depth  review  of  the  adequacy  and 
viability of the services provided by Europeana as a Digital Service Infrastructure under 
the CEF programme. 

This report summarises the outcomes of the public online consultation and of the other 
consultations conducted by the Commission to ensure that the opinion of individuals and 
organisations is well reflected in the evaluation as well as in the Commission’s proposal 
for Europeana’s future. 

2.  PUBLIC CONSULTATION 

The public online consultation was held, in all EU languages, between 17 October 2017 
and 14 January 2018. 

All  organisations  and  members  of  the  public  with a  personal  or  professional  interest  in 
digital  cultural  heritage,  or  in  Europeana  in  particular,  were  invited  to  express  their 
opinion,  experience  and  expectations  on  Europeana  and  the  various  aspects  of  the 
platform. 

A targeted questionnaire was included in the public online consultation. It was addressed 
to  professionals  and  organisations  working  in  cultural  heritage,  research,  education  and 
the creative sector that had already used Europeana as a platform for sharing or re-using 
cultural heritage content. 

Respondents profile 

Of  the  1 22114  respondents  that  took  part  in  the  online  consultation,  883  responded  as 
individuals (72 %) and 338 responded on behalf of an organisation (28 %). 

Culture,  research  and  education  were  the  most  represented  sectors  for  both 
individuals  and  organisations.  Museums  and  libraries  were  the  most  represented  culture 
organisations,  together  covering  more  than  70 %  of  respondents  from  that  field.  Some 
63 %  of  those  responding  on  behalf  of  educational  organisations  represented  higher 
education. 

Among individuals, 66 % indicated that their main interest in Europeana was for personal 
use and 61 % for professional use. 

13 Council Conclusions of 31/5/2016 on the role of Europeana for the digital access, visibility and use of 

European cultural heritage (OJ C 212, 14.6.2016, p. 9). 

14 Overall, the online consultation received 1 226 responses, of which 5 duplicates were deleted during the 

clean-up process. The resulting dataset consists of 1 221 responses. 

17 

 
                                                            
Use of Europeana platform — main results 

Respondents  agree  that  the  portal  ‘Europeana  Collections’  (europeana.eu)  is  valuable  in 
various  aspects:  it  advances  cultural  knowledge  and  learning  (82 %  of  respondents); 
Thematic  collections  and  exhibitions  allow  exploration  of  specialised  content  (85 %); 
Trustworthiness of content sources (82 %); Availability of material of different languages 
(79 %).  

Two  thirds  of  respondents  indicated  that  Europeana  was  important  for  finding  and 
exploring European cultural heritage. 

Respondents  also  agree  that  Europeana  is  relevant  for  those  interested  in  using  digital 
cultural  heritage  material  in  their  work,  particularly  in  providing  relevant  material  with 
suitable quality for re-use (75 % of respondents), suitable licencing for re-use (65 %) and 
assuring the trustworthiness of its data (75 %). 

Respondents  replying  as  individuals  were  asked  how  often  they  used  different  parts  of 
Europeana. 35 % indicated that they use the portal ‘Europeana Collections’ on at least a 
monthly  basis.  Similarly,  20 %,  15 %,  and  8 %  indicated  that  they  use  the  professional 
pages  of  Europeana,  Europeana  material  for  other  projects,  and  Europeana’s  APIs, 
respectively, on at least a monthly basis. At most 8 %, at most, indicated that they use the 
Europeana platform on a weekly basis (‘Every day’ or ‘A few times per week’). 

Respondents  replying  as  individuals  were  asked  what  other  portals  they  use  to  access 
cultural  heritage  information.  Overall,  58 %  of  the  respondents  use  Wikipedia  on  a 
weekly basis (‘Every day’ or ‘A few times per week’), followed by the portals of national 
institutions (22 %) and the Internet Archive (21 %)  

Out  of  201  respondents  that  elaborated  on  the  aspects  they  particularly  like  in  cultural 
heritage  portals  which  they  could  not  find  on  Europeana  (i) 19  stated  that  they  seek 
higher or consistent quality of content and another 20 that they seek content pertaining to 
specific  themes;  (ii) 69  pointed  to  a  better  search  functionality,  especially  on  the 
relevance  of  the  search  hits;  (iii)  33  pointed to  the  quality  of  the  metadata,  such  as  the 
level of detail of the descriptions; and (iv) 12 respondents pointed to clearer, as well as 
less restrictive licencing. 

On  Europeana’s  campaigns  and  activities  to  engage  citizens  with  culture,  among  the 
individuals  that  responded  to  the  consultation,  9 %  of  respondents  were  aware  and  had 
engaged with the ‘Collection days’ campaign, while  49 % were  aware of the campaign 
but did not engage. Apart from this campaign, the majority (65 %) of respondents were 
unaware of other Europeana campaigns and activities.  

User experience with Europeana — main results 

Questions related to the user experience with the Europeana platform were addressed to 
respondents replying as individuals. 

The  majority  indicated  that  they  seek  images  on  Europeana  (88 %),  followed  by  texts 
(64 %). 59 % use English for their searches. 41 % indicated that they often find what they 
search for while another 44 % indicated that they sometimes find what they search for. 

In  relation  to  the  user  experience  when  searching  on  Europeana  the  following 
information was obtained.  

18 

 
  57 %  of  the  respondents  (452  out  of  794)  were  satisfied  with  the  ease  of  searching 
and filtering results, 48 % with the ease of navigating through multilingual material, 
and 53 % with the relevance and accuracy of results. 

  45 %  of  the  respondents  (182  out  of  404)  were  satisfied  with  the  ease  of  using  the 

APIs. 

  69  respondents  described  problems  when  searching  on  Europeana.  They  pointed 
mainly to a lack of website features, such as the level of multilingualism, the lack of 
sophisticated filtering for refining search results, a need for more advanced searching 
possibilities,  and  a  lack  of  relevance  of  the  search  results  (inconsistencies,  broken 
links, lack of thumbnails and duplicates or redundant hits in the search results). 

In terms of satisfaction with various aspects of the content and metadata: 

 

the  percentage  of  respondents  that  indicated  satisfaction  with  the  content  on 
Europeana  ranged  between  50 %  and  63 %  in  terms  of:  (i) geographical  coverage, 
(ii) thematic  coverage,  (iii)  quantity,  (iv) technical  quality,  (v)  uniqueness,  and  (vi) 
re-usability; 

  71  respondents  provided  constructive  criticism  about  Europeana’s  content,  pointing 

mainly to the content quantity, quality, and licencing; 

  The percentage of respondents that indicated satisfaction with Europeana’s metadata 
terms  of:  (i) completeness,  (ii) detail, 
ranged  between  52 %  and  62 % 
in 
(iii) accuracy,  (iv) relevance,  and  (v) links  to  original  objects.  Only  46 %  indicated 
satisfaction  with  the  multilingualism  of  metadata  (availability  of  metadata  in  their 
language); 

  65 respondents provided constructive criticism regarding issues on the availability of 

metadata in multiple languages, metadata quality and consistency; 

Data partners 

310  respondents,  replying  on  behalf  of  an  organisation  or  as  individuals  with  a 
professional interest in Europeana, had collaborated with Europeana as a data partner and 
took  part  in  this  targeted  section.  Most  respondents  believe  it  is  important  to  provide 
content to Europeana and indicated that they were likely to continue supplying data via an 
aggregator (84 %) or directly (58 %) in the future. Lack of time or resources (31 %) and 
lack of information on how to provide data (25 %) were indicated as the largest barriers to 
providing content. 

The highest proportion of data partners (82 %) agree that Europeana’s promotion of open 
culture/metadata and content has facilitated their work within their organisation, while at 
least  two  thirds  of  data  partners  indicated  that  Europeana  generated  value  for  their 
organisation via audience reach (72 %), added value to content (74 %), and facilitation of 
partnerships with other cultural institutions (65 %). 

The  percentage  of  data  partners  indicating  satisfaction  with  Europeana’s  aggregation 
structure ranged between 35 % and 44 % in terms of:  (i) the time it takes from source to 
publication, (ii) the level of automation along the aggregation chain, (iii) communication, 
(iv) the  quality  assurance  process,  (v)  support,  and  (vi) overall  effort  to  provide  data. 
Some  40  respondents  elaborated  on  their  answer  in  text.  Negative  responses  mainly 
highlighted  a  lack  of  tools  in  the  aggregation  process,  notably  for  content  providers  to 
easily  publish  and  update  their  content,  as  well  as  a  lack  of  fluidity  of  communication 
between  Europeana  and  the  aggregators  or  the  providers  in  order  to  resolve  issues  and 
facilitate  the  publication  of  data.  Of  the  positive  responses,  the  main  point  highlighted 

19 

 
was  satisfaction  with  the  technical  support  provided  by  aggregators  to  the  content 
providers. 

Re-use of material 

89 respondents replying on behalf of an organisation and 534 replying as individuals, 
who indicated that they had, at least once or twice, re-used material from Europeana, took 
part in the targeted section on re-use. The highest proportion of respondents indicated 
that they re-used content from Europeana in research projects or as teaching materials. 

72  respondents  elaborated  in  text  on  how  they  have  re-used  the  material.  Four  main 
themes  emerged:  in  presentations  (17  respondents);  in  school  or  academia,  such  as  in 
student  assignments  (15  respondents);  on  the  web,  e.g.  in  blogs,  social  media,  and  the 
integration  of  the  Europeana  API  into  other  portals  (13  respondents);  and  for  personal 
recreation (8 respondents). 

In  relation to awareness of Europeana’s  activities  on engaging re-use  professionals, the 
most  well-known  activity  was  the  Europeana  hackathons.  221  respondents  (52 %) 
indicated  that  they  were  aware  of  such  activity,  although  only  a  small  proportion  (less 
than  7 %)  indicated  that  they  engaged  in  a  hackathon.  Most  respondents  claimed  to  be 
unaware of other activities. 

Europeana Network Association (ENA) 

222  respondents  took  part  in  the  targeted  section  regarding  engagement  with  the 
activities of the ENA, replying as individuals or on behalf of an organisation. Of these, 
about two thirds indicated that they participate at least occasionally in the elections for the 
Members’ Council, in the Annual General Meetings and in working groups or task forces. 

Respondents  indicated  that  the  ENA  has  enabled  them  to  contact  professionals  and 
experts they would not otherwise be able to find (71 %), that such exchanges have been 
beneficial to them (70 %), and that the ENA complements other digital cultural heritage 
networks (72 %). More than half indicated that it is easy to interact with ENA members 
and that they feel they had a say in how Europeana is developed. 

14 out of the 60 respondents that provided comments about participating in the ENA felt 
that their contributions were not taken into account by Europeana decision makers. 

EU added value 

At  least  two  thirds  of  the  respondents  agreed  that  Europeana  brings  value  to  the  EU, 
particularly  in  digitally  bringing  together  and  providing  access  to  cultural  objects  from 
across  Europe  (85 %),  standardising  the  format  of  object  descriptions  (70 %),  and 
facilitating a European network of digital heritage professionals (70 %). 

30  respondents  elaborated  on  ways  that  Europeana  has  created  value  on  the  European 
level.  The main  positive  topic  raised  was  that Europeana  had  underpinned  a  sense  of  a 
shared history and identity among European citizens. 

158  respondents  elaborated  on  national,  European,  or  international  developments  that 
they thought would not have happened had Europeana not existed, such as: 

  national  or  institutional  level  initiatives,  such  as  the  creation  of  online  cultural 
heritage  portals  based  on  Europeana’s  model,  or  initiatives  that  aggregate  and 
provide content to Europeana (11 %); 

20 

 
  European  developments:  Europeana’s  contribution  to  the  digitisation  and  online 
accessibility  of  content  (6 %)  and  improved  cooperation  within  the  cultural 
heritage sector (18 %); 
 
international collaborative efforts to which Europeana has contributed (2 %); 
  other initiatives, which have been enabled thanks to Europeana’s efforts such as 
the establishment of licencing standards and liberalisation of licences (8 %). 

Forward looking questions 

In the last section of the consultation, 585 respondents expressed their views on the most 
significant issues and priorities for Europeana. The following topics emerged. 

  Website functionality: 

  dissatisfaction with Europeana’s current search function, for instance hits with 

low relevance, limited filtering of search results (14 %); 

  higher level of multilingualism on the Europeana website; metadata provided 

not only in the native language (7 %). 

  Communication activities: 

  more  efforts  to  raise  awareness  and  visibility  of  Europeana  to  the  general 

public, as well as to teachers, students and researchers (21 %). 

  Content on Europeana: 

  quantity: wish for more content from underrepresented countries, of specific 

themes and different formats (15 %); 

  quality: more quality content, such as images in higher resolution and without 

watermarks (7 %). 

  Licencing of the content — respondents indicated a wish for: 

  Europeana to urge content providers to make their content available under less 

restrictive licencing, particularly for non-commercial use (8 %); 

  greater  clarity  and  accuracy  in  the  re-use  declarations  associated  with  the 

content on Europeana (4 %). 

  Metadata of Europeana’s content — respondents wish for: 

  semantic enrichment of the metadata, better linking of the data, and stronger 

enforcement of metadata standards (6 %); 
  complete, accurate and detailed metadata (5 %). 

  Aggregation and data ingestion: 

 

respondents  indicated  a  wish  for  more  technical  support  to  data  providers 
(5 %). 

395  respondents  expressed  views  on  what  new  features,  services,  or  activities  could  be 
introduced by Europeana in the future, including: 

 
 

 

 
 

in-text search and advanced filtering (13 %); 
tools  such  as  smartphone  apps,  download  facilities,  virtual  exhibition  creators, 
and games (7 %); 
tools  allowing  users  to  contribute  to  Europeana  by  correcting  metadata,  social 
tagging, or commenting (5 %); 
improved multilingualism by machine translation (4 %); and 
tools for data providers to upload content and update metadata (5 %). 

21 

 
3.  OTHER STAKEHOLDER CONSULTATION ACTIONS 

In addition to the online consultation, the following consultation actions were carried out: 

  During a four-week period, all interested stakeholders were able to provide feedback 
on  the  published  Evaluation  Roadmap15,  outlining  the  overall  evaluation  approach. 
The  Europeana  Foundation,  the  Europeana  Network  Association  and  the  Flemish 
government submitted comments. 

  Eight case study interviews were conducted under the evaluation study, on: 

o  ‘The value of the Europeana Network’; 
o  ‘Benefits for cultural institutions that share data in Europeana’; 
o  ‘Europeana Publishing Framework (EPF) — activities and impact’; and 
o  ‘Re-use of Europeana in education’. 

The  input  received  through  these  interviews  confirmed  the  perceived  value  of 
Europeana  and  its  network  of  aggregators  and  cultural  institutions  for  the 
interoperability  and  online  accessibility  of  cultural  material  as  well  as  for  the 
development  of  standards  and  common  solutions.  Visibility,  and  recognition  and 
being part of a larger ecosystem were mentioned as the main motivations for cultural 
institutions to share data in Europeana. 

Interviews pointed out that it is important to get the data providers more involved, by 
developing the infrastructure to facilitate data provision and reduce the complex and 
time-consuming  workflows  for  publication.  Continuous  support  provided  by 
Europeana’s  national  and/or  domain  aggregators  and  expert  hubs  was  perceived  as 
very  important  in  order  to  reach  and  support  data  providers,  particularly  small 
institutions. 

Interviews  also  highlighted  an 
the  problem  of 
multilingualism. Some cultural institutions do not have the capacity to share metadata 
in  a  language  other  than  their  native  language,  which  negatively  affects  the  quality 
and findability of the material. 

increased  awareness  of 

Efforts  to  explore  re-use  of  Europeana  collections  in  education  —  either  integrated 
into normal teaching in the classroom, or through integration in national educational 
resources portals to contribute the European perspective — showed promising results 
and good potential to have a larger impact. However, the problems of finding suitable 
content in the appropriate language were reported as hindering wider integration and 
the re-use of Europeana collections in education. 

  As an extension to the case study on benefits for cultural institutions that share data 
in  Europeana,  a  short  survey  was  completed  by  nine  small  cultural  institutions  in 
Belgium,  Germany  and  Italy.  Two  national  institutions  in  Luxembourg  were  also 
invited  to  complete  this  short  survey,  since  no  organisation  from  Luxembourg 
participated  in  the  online  consultation.  Input  received  through  these  questionnaires 
coincided  with  the  input  collected  through  the  interviews  for  this  case  study.

15 http://ec.europa.eu/smart-regulation/roadmaps/docs/plan_2016_55_europeana_evaluation_en.pdf 

22 

 
 
 
 
 
 
 
 
                                                            
ANNEX 3: EVALUATION QUESTIONS 

The  evaluation  roadmap  set  out  the  following  questions  to  be  addressed  by  the 
evaluation. The evaluation questions were validated during the evaluation process. 

Relevance 

  To  what  extent  is  the  concept  of  Europeana  relevant  to  the  needs  of  the  EU?  Are 
Europeana’s outputs and results relevant to EU priorities, in particular the flowering 
of cultures of the Member States (Article 167 TFEU) and the Digital Single Market? 
  Are Europeana’s services relevant to the needs of its user groups: data partners, end 

users, re-users? 

  How relevant is the content and metadata? 
  How  relevant  is  Europeana  as  a  facilitator  of  professional  exchange  of  cultural  and 

technological expertise in the digital heritage community? 

Effectiveness 

  How  successful  have  the  offered  services  been  in  meeting  requirements  of  the 
different audiences? How effective is Europeana in reaching end-users and re-users? 
  What has been the progress towards achieving an impact, based, where applicable, on 
statistics  and  performance  indicators?  Are  the  current  performance  indicators  of 
Europeana sufficient for monitoring progress? 

  Has  the  underlying  technical  infrastructure,  data  model  and  functionality  of 

Europeana been adequate for achieving its objectives? 

  Does Europeana’s governance ensure the most inclusive and effective operation? 
  To  what  extent  have  the  presentation  of  results  and  outputs  of  Europeana  to 

stakeholders and the public been effective? 

  How does Europeana meet the requirements of a fast changing technological and user 

environment? 

Efficiency 

  How efficient is the current aggregation infrastructure? 
  How efficient is the current governance structure? 
  To  what  extent  is  the  current  support  scheme  efficient  to  respond  to  the  needs  of 

Europeana and its stakeholders? 

  How do the costs involved compare to the achieved results and impact? 

Coherence 

  How coherent are Europeana’s activities and services internally? 
  To what extent is Europeana coherent with national strategies in Member States and 
with  other  national  or  international  initiatives  with  similar  objectives  (e.g.  national 
aggregators  such  as  Gallica,  Hispana  or  German  Digital  Library;  Research 
infrastructures such as DARIAH or E-RIHS; or DPLA) 

  To  what  extent  is  Europeana  coherent  with  wider  EU  policy,  including  with  the 
European Agenda for Culture in 2007 whereby cultural heritage has been a priority 
under successive Council Work Plans for Culture? 

23 

 
 
EU added value 

  What is the EU added value of Europeana compared to what could be achieved by the 
private sector or by Member States at national, regional and/or institutional level? 
  Have  there  been  national,  European  or  international  developments  that  would  not 

have happened without Europeana? 

24