Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions - Final evaluation of the results obtained in implementing the multiannual Community programme to stimulate the development and use of European digital content on the global networks and to promote linguistic diversity in the information society (eContent)
Date: 2007-01-29
Institutions: European Commission
Subjects: EU programme, Internet, access to information, information society, information technology, multilingualism, project evaluation

COMMISSION OF THE EUROPEAN COMMUNITIES 

Brussels, 29.1.2007 
COM(2007)29 final 

COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN 
PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL 
COMMITTEE AND THE COMMITTEE OF THE REGIONS 

Final evaluation  
of the results obtained in implementing the multiannual Community programme to 
stimulate the development and use of European digital content on the global networks 
and to promote linguistic diversity in the information society (eContent) 

EN 

   EN 

 
 
 
  
COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN 
PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL 
COMMITTEE AND THE COMMITTEE OF THE REGIONS 

Final evaluation  
of the results obtained in implementing the multiannual Community programme to 
stimulate the development and use of European digital content on the global networks 
and to promote linguistic diversity in the information society (eContent) 

1. 

INTRODUCTION 

This Communication concerns the final evaluation of the eContent programme 2001-
2005, subject to a mid-term evaluation1 in 2003 with a view to assessing its first two 
years of implementation. This final evaluation2, covering the period 2003-2004, was 
carried out by a panel of 3 independent experts, supported by a team of professional 
evaluators from a company3 in charge of structuring and supporting the evaluation, 
collecting  evidence  and  drafting  the  report.  The  evaluation  was  conducted  from 
February  to  May  2006,  in  close  collaboration  with  the  Commission  services 
concerned4.  

2. 

BACKGROUND 

The aim of the eContent programme (hereinafter referred to as “the Programme”), as 
specified  in  the  Council  Decision5,  was  to  stimulate  the  development  and  use  of 
European digital content on the global networks and to promote linguistic diversity 
in the information society. The programme ran for four years from 18 January 2001 
with a budget of 100 million euro. 

The Programme addressed three distinct Action Lines: 

(1) 

(2) 

improving access to and expanding use of public sector information; 

enhancing  content  production 
environment; 

in  a  multilingual  and  multicultural 

(3) 

increasing dynamism of the digital content market. 

It was implemented through cost-shared projects selected on the basis of public calls 
for proposals. 

1 

2 

3 

4 
5 

Mid term evaluation of the eContent Programme: Final Report, June 2003, Technopolis Ltd., IDATE, 
PRISMA, full report available at www.cordis.lu/econtent 
Final  Evaluation  of  the  eContent  Programme  (2001-2005):  Final  Report,  June 2006, RAND  Europe, 
http://ec.europa.eu/information_society/activities/econtentplus/index_en.htm. 
RAND Europe, contracted on the basis of a restricted call for tenders issued by DG Information Society 
in the autumn of 2005. 
DG INFSO Units C3 and E6 
OJ L 14, 18.1.2001, p.32 

EN 

2 

   EN 

 
                                                 
the  period  2003-2004,  64  projects  were  co-funded 

During 
involving  404 
organisations  and  covering  24  Member  States  plus  Iceland,  Norway  and  Bulgaria; 
Romania; Turkey. The total expenditure under the 2003 and 2004 budgets, including 
the funding of all cost-shared projects, the costs of evaluations, reviews and meetings 
was around 56 million euro.  

3. 

EVALUATION OBJECTIVES 

The  evaluation  assessed  the  following  issues:  relevance  of  the  programme's 
objectives,  priorities  and  means  of  implementation,  the  effectiveness  and  impact  of 
the  programme,  its  efficiency  and  cost  effectiveness,  its  utility  and  sustainability, 
causal  links  from  resources  used  through  to  activities  and  presumed  impacts  (the 
intervention  logic)  and  lessons  to  be  learnt  for  possible  future  interventions  of  a 
similar type. 

4. 

EVALUATION FINDINGS 

The  evaluators  found  that  the  programme  was  successfully  implemented  and  the 
attribution  of  the  grants  and  the  management  of  the  programme  were  in  general 
transparent, efficient and satisfactory. 

More specifically, the evaluators made the following four major findings: 

Finding 1: Solid Performance of Basic Functions 

At  the  level  of  activities  and  of  results  the  performance  of  eContent  was  good:  the 
Programme  made  real  and  substantial  contributions  to  expanding  the  use  of  public 
sector  information,  and  to  multi-lingual  and  multi-cultural  content  production. 
Overall 
the  programme  produced  beneficial  results 
commensurate  with  its  resources;  that  it  was  cost  effective  in  the  generation  of  its 
outputs  and  results,  and  that  it  stimulated  activities  that  would  either  not  otherwise 
have  happened  at  all,  or  would  only  have  occurred  at  a  national  level.  The  overall 
management of the programme was good, and processes were fair and transparent.  

it  was  concluded 

that 

Finding 2: Adaptation to Rapid Change in the Sector 

The  exceptionally  rapid  development  of  the  sector  substantially  affected  the 
objectives  of  the  programme:  in  particular,  the  original  intention  of  ‘kick-starting’ 
certain classes of activity was achieved much more quickly than could initially have 
been anticipated. The programme responded well to this change of circumstance with 
a  mid-term  modification  of  its  objectives  (expressed  as  modification  to  its  Action 
Lines) and the evaluators welcomed this response. That change in its turn produced a 
need,  however,  for  a  greater  focus  on  the  development  of  networking;  community 
building  and  dissemination  at  programme  level  at  a  rather  late  stage  in  the 
programme’s life cycle. The evaluators considered that the horizontal elements of the 
programme  response  to  this  need  were  of  sufficient  quality  but  not  of  sufficient 
quantity. 

EN 

3 

   EN 

 
Finding 3: Programme Level Added Value 

In  respect  of  progress  towards  the  programme’s  wider  socio-economic  objectives, 
there is a distinction between the evaluator’s conclusions regarding eContent as a set 
of projects (where objectives in the three action lines are expected ultimately to be 
achieved)  and  conclusions  about  eContent  as  a  synergistic  whole  (where  the  Panel 
found  insufficient  indication  of  the  development  of  programme  level  synergies 
sufficient to create significant added value over and above the combined value of the 
individual projects). The evaluators considered that the value of networking between 
projects had not been sufficiently recognised and that dissemination and concertation 
activities  at  programme  level  had  been  insufficient.  The  relatively  modest  scale  of 
eContent’s intervention was thus not leveraged to the greatest possible extent. 

Finding 4: Relationship with Current and Future Programmes 

The  programme  was  seen  to  have  achieved  good  and  effective  penetration  into  the 
Accession  countries;  to  have  complemented  the  e-TEN  initiative  well  and  to  have 
been well aligned with the eEurope Action Plan, although synergies with Structural 
Fund programme were not apparent to the evaluators. The evaluators considered that 
this  offers  useful  and  directly  applicable  lessons  for  the  eContent  Plus  programme 
and  to  the  CIP  ICT  Policy  Support  Programme  (CIP-ICT).  Lessons  learned  in  this 
respect  (e.g.  concerning  the  particular  requirements  of  community  building, 
networking, dissemination and exploitation) have the potential to create mechanisms 
fostering  a  substantial  contribution  to  the  development  of  long-term  sustainable 
socio-economic benefits foreseen in the Lisbon and Gothenburg agendas. 

5. 

RECOMMENDATIONS AND COMMISSION COMMENTS 

The  above  major  findings  give  rise  to  the  following  recommendations  for  future 
programmes and comments by the Commission: 

Recommendation  1:  The  evaluators  recommended  that  future  near-to-market 
programmes  should  devote  a  very  much  greater  proportion  of  their  resources  to 
networking and concertation between projects and to dissemination of the exemplary 
elements  of  project  activity  that  projects  themselves  may  have  no  incentive  to 
disseminate.  It  is  also  recommended  that  consideration  be  given  to  support  for 
programme events and conferences. 

Commission  comments:  To  foster  networking  between  projects  and  better  focus 
dissemination at programme level, Thematic Networks have been introduced on the 
main actions of the eContentplus programme. They serve as an instrument to bring 
together relevant stakeholders, expertise and facilities to support awareness-raising 
activities and the identification and development of opportunities for improving the 
availability  of  digital  content  and  the  co-ordination  of  projects  funded  under  the 
programme. 

Recommendation  2:  Two  areas  were  identified  where  studies  might  usefully  be 
undertaken  for  the  benefit  of  future  near-to-market  programmes.  More  detailed 
understanding  of  the  modalities  of  the  diffusion  and  dissemination  of  results  from 
near-to-market projects and programmes is needed in order to maximise programme-

EN 

4 

   EN 

 
level  added  value;  and  there  is  a  need  for  the  development  of  tools  that  would  be 
useful in assessing movement towards achieving the long-term outcomes of near-to-
market programmes. 

Commission comments: The Commission is planning to launch a number of studies 
in the framework of the follow-on programme eContentplus, e.g. on investment in the 
digitisation  of  cultural  resources  and  the  economic  impact  of  the  preservation  of 
digital resources. 

Recommendation  3:  It  is  further  recommended  that  Programmes  devote  some 
resources to monitoring the long-term development of market outcomes as part of an 
ongoing  engagement  with  the  target  sector  (in  this  case,  European  eContent).  This 
would facilitate evaluation; raise awareness of interventions and project outputs, and 
assist the “fine-tuning” essential in connection with rapidly evolving sectors. 

Commission  comments:  These  recommendations  are  already  addressed  by  the 
Commission  through  a  number  of  activities:  Data  on  the  Information  Market 
available  in  the  public  domain  on  the  size  and  growth  of  the  market  are  currently 
collected and structured. 

6. 

CONCLUSION 

The  Commission  takes  full  note  of  the  findings  of  the  final  evaluation  of  the 
programme  and  will  take  account  of  the  recommendations  in  implementing  the 
eContentplus  programme  and  in  planning  a  future  follow-up  programmes.  Progress 
already made in the area mentioned by the recommendations will be reinforced. 

The Commission considers that the eContent programme 

(1) 

has been successfully implemented ; 

(2) 

(3) 

enhanced  the  role  of  digital  content  as  a  means  to  increase  access  to 
information  by  the  citizens  and  to  stimulate  the  economic  and  social 
development of European businesses ; 

promoted  rich  and  diverse  digital  content  and  services  contributing  to  the 
creation of a single European information space and an Information Society 
for growth and employment. 

EN 

5 

   EN 

 
Final Evaluation of the eContent programme (2001-2005): 
Final Report, June 2006, RAND Europe 

Executive Summary 

Background 

The eContent programme of 2001 to 2005 was one of a series of programmes developed in 
order to stimulate the growth and development of the digital content sector in Europe. Its core 
consideration  was  that  European  enterprises  and  European  citizens  should  be  enabled  to 
access and to use high quality digital content of European origin in a manner appropriate to 
their  needs.  The  wider  context  for  the  programme  included  the  integration  of  the  (then) 
candidate  countries  (many  of  which  are  now  Member  States  of  the  European  Community); 
and other evolving European policy objectives including particularly the Lisbon objectives. 

Focus of the Final Evaluation 

The programme’s total budget of 100MEURO has been fully committed and its activity very 
largely  completed,  triggering  the  requirement  for  this  ex-post  Final  Evaluation.  An  Interim 
Evaluation was undertaken in 2003 i.e. at about the mid-point of the programme, but this was 
too early to consider the contribution of the visible results and outputs of engaged projects to 
wider  socio-economic  outcomes  established  as  the  objectives  of  the  programme.  Such 
outcomes necessarily require time to develop (and indeed it is still too soon to expect them to 
be  fully  visible);  a  large  part  of  this  evaluation  is  accordingly  concerned  with  the  extent  to 
which  the  programme,  and  the  interventions  within  it,  produced  or  were  consonant  with 
mechanisms  capable  of  producing  the  required  outcomes.  Issues  of  dissemination  and  of 
exploitation and the relationships established between the programme and the sector in which 
it operated are central to consideration of these mechanisms. 

The  full  report  of  the  evaluation  sets  out  its  findings  in  detail.  Key  findings  include  the 
following: 

Finding 1: Solid Performance of Basic Functions 

At the level of activities and of results the performance of eContent was good: the Programme 
made real and substantial contributions to expanding the use of public sector information, and 
to  multi-lingual  and  multi-cultural  content  production.  Overall  it  was  concluded  that  the 
programme  produced  beneficial  results  commensurate  with  its  resources;  that  it  was  cost 
effective in the generation of its outputs and results, and that it stimulated activities that would 
either not otherwise have happened at all, or would only have occurred at a national level. The 
overall management of the programme was good, and processes were fair and transparent. 

Finding 2: Adaptation to Rapid Change in the Sector 

The exceptionally rapid development of the sector substantially affected the objectives of the 
programme:  in  particular,  the  original  intention  of  ‘kick-starting’  certain  classes  of  activity 
was achieved much more quickly than could initially have been anticipated. The programme 
responded well to this change of circumstance with a mid-term modification of its objectives 
(expressed  as  modification  to  its  Action  Lines)  and  the  evaluators  welcomed  this  response. 
That change in its turn produced a need, however, for a greater focus on the development of 
networking; community building and dissemination at programme level at a rather late stage 

EN 

6 

   EN 

 
in the programme’s life cycle. The evaluators considered that the horizontal elements of the 
programme response to this need were of sufficient quality but not of sufficient quantity.  

Finding 3: Programme Level Added Value 

In respect of progress towards the programme’s wider socio-economic objectives, there is a 
distinction between the evaluator’s conclusions regarding eContent as a set of projects (where 
objectives  in  the  three  action  lines  are  expected  ultimately  to  be  achieved)  and  conclusions 
about  eContent  as  a  synergistic  whole  (where  the  Panel  found  insufficient  indication  of  the 
development  of  programme  level  synergies  sufficient  to  create  significant  added  value  over 
and above the combined value of the individual projects). The evaluators considered that the 
value  of  networking  between  projects  had  not  been  sufficiently  recognised  and  that 
dissemination  and  concertation  activities  at  programme  level  had  been  insufficient.  The 
relatively  modest  scale  of  eContent’s  intervention  was  thus  not  leveraged  to  the  greatest 
possible extent. 

Finding 4: Relationship with Current and Future Programmes 

The programme was seen to have achieved good and effective penetration into the Accession 
countries; to have complemented the e-TEN initiative well and to have been well aligned with 
the  eEurope  Action  Plan,  although  synergies  with  Structural  Fund  programme  were  not 
apparent  to  the  evaluators.  The  evaluators  considered  that  this  offers  useful  and  directly 
applicable  lessons  for  the  eContent  Plus  programme  and  to  the  CIP  ICT  Policy  Support 
Programme  (CIP-ICT).  Lessons  learned  in  this  respect  (e.g.  concerning  the  particular 
requirements  of  community  building,  networking,  dissemination  and  exploitation)  have  the 
potential  to  create  mechanisms  fostering  a  substantial  contribution  to  the  development  of 
long-term  sustainable  socio-economic  benefits  foreseen  in  the  Lisbon  and  Gothenburg 
agendas. 

Recommendations 

The  evaluators  recommended  that  future  near-to-market  programmes  should  devote  a  very 
much  greater  proportion  of  their  resources  to  networking  and  concertation  between  projects 
(for  instance,  to  encourage  the  development  and  diffusion  of  new  business  models)  and  to 
dissemination of the exemplary elements of project activity that projects themselves may have 
no incentive to disseminate. It is also recommended that consideration be given to support for 
programme events and conferences, to the operation of a press office and for full involvement 
of all appropriate media particularly including search engine optimisation, viral marketing etc 
whilst  still  utilising  trade  and  specialist  press.  Such  horizontal  activities  are  vital  to  the 
optimal development of the potential value of a programme and therefore should not be left 
hostage to the necessary uncertainties of the ‘Call for Proposal’ mechanism. 

Two areas were identified where studies might usefully be undertaken for the benefit of future 
near-to-market  programmes.  More  detailed  understanding  of  the  modalities  of  the  diffusion 
and dissemination of results from near-to-market projects and programmes is needed in order 
to maximise programme-level added value; and there is a need for the development of tools 
(analogous to the citation analysis tools used in assessing the outcomes of RTD programmes) 
that  would  be  useful  in  assessing  movement  towards  achieving  the  long-term  outcomes  of 
near-to-market programmes. 

EN 

7 

   EN 

 
It is further recommended that Programmes devote some resource to monitoring the long-term 
development of market outcomes as part of an ongoing engagement with the target sector (in 
this  case,  European  eContent).  This  would  facilitate  evaluation;  raise  awareness  of 
interventions  and  project  outputs,  and  assist  the  “fine-tuning”  essential  in  connection  with 
rapidly evolving sectors. 

EN 

8 

   EN