REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Final evaluation of the Eurostars Joint Programme (2008-2013)
Date: 2015-09-30
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU financing, EU programme, EU research policy, aid to undertakings, project evaluation, report, research and development, small and medium-sized enterprises

EUROPEAN 
COMMISSION 

Brussels, 30.9.2015  
COM(2015) 479 final 

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL 

Final evaluation of the Eurostars Joint Programme (2008-2013) 

EN 

   EN 

 
 
 
 
  
 
1. 

INTRODUCTION 

In July 2008, the European Parliament and the Council adopted a decision providing for EU 
participation  in  financing  the  Eurostars  Joint  Programme  (‘Eurostars’)1.  The  Eurostars 
Decision limited the EU financial contribution to ‘the equivalent of a maximum of one third 
of  the  effective  contributions  of  the  participating  Member  States  and  the  other  participating 
countries,2 within a ceiling of EUR 100 million.’ The participation was for the duration of the 
Seventh  Framework  Programme  for  Research  (2008-2013).  Since  then,  the  European 
Parliament  and  the  Council  have  adopted  a  new  Decision  providing  for  continued 
participation  in  funding  in  the  new  Eurostars-2  joint  programme  from  2014  to  2020 
(Eurostars-2 Decision).3  

Eurostars aims to provide financial support to transnational market-oriented research projects 
initiated  and  led  by  R&D-performing4  small  and  medium-sized  enterprises  (SMEs).5  These 
firms should be able to carry out the greater part of a project’s R&D work, and should be able 
to  exploit  its  results  commercially,  thus  improving  their  competitive  position.  Eurostars 
projects are collaborative, meaning that in any project, there should be at least two partners 
(autonomous legal entities) from two different participating states, with at least one being an 
R&D-performing  SME.  Projects  must  have  a  maximum  duration  of  three  years,  and  within 
two years of project completion, the product of the research should be ready for launch onto 
the market. 

The EUREKA6 Secretariat, based in Brussels, acts as the central support unit for the network 
and  is  the  dedicated  implementation  structure  for  the  programme.  The  Secretariat  organises 
calls for proposals, verifies the eligibility of applications and selects projects for funding. It is 

1   Decision  No  743/2008/EC  of  the  European  Parliament  and  of  the  Council  of  9  July  2008  on  the 
Community’s participation in a research and development programme undertaken by several Member States 
aimed at supporting research and development performing small and medium-sized enterprises. (OJ L 201, 
30.07.2008, p.58-67): 

      http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2008:201:0058:0067:EN:PDF. 
2   All  28  Member  States  plus  five  other  countries  (Iceland,  Israel,  Norway,  Switzerland,  and  Turkey)  have 
participated in Eurostars (2008-2013). As of February 2008, 20 Member States had started their participation. 
Belgium, Croatia, Italy, Latvia, Luxembourg and United Kingdom have participated since November 2008, 
Bulgaria since February 2010, and Malta since March 2011. 

3   Decision  No  553/2014/EU  of  the  European  Parliament  and  of  the  Council  of  15  May  2014  on  the 
participation of the Union in a research and development programme jointly undertaken by several Member 
States aimed at supporting research and development-performing small and medium-sized enterprises (OJ L 
169, 07.06.2014, p. 1-13): 
http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014D0553&from=EN. 

4   Under Eurostars 2008-2013, R&D-performing SMEs are defined as SMEs dedicating at least 10 % of their 

staff (full-time equivalents), or 10 % of their turnover, to R&D activities. 

5   The definition of SMEs is set out in Recommendation 2003/361/EC of 6 May 2003 concerning the definition 

of micro, small and medium-sized enterprises (OJ L 124, 20.05.2003, p.36): 

      http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2003:124:0036:0041:EN:PDF. 

The  main 
factors determining whether a company is an SME are the number of employees (<250) and either turnover 
(<€50 m) or balance sheet total (<€43 m). 

6   EUREKA  is  an  intergovernmental  network  launched  in  1985  to  support  market-oriented  R&D  and 
innovation  projects  by  industry,  research  centres  and  universities  across  all  technological  sectors.  It  is 
composed  of  41  members, 
the  Commission: 
http://www.eurekanetwork.org/. 

the  European  Union 

represented  by 

including 

2 

 
                                                            
also responsible for allocating  the EU financial contribution. National funding bodies in the 
participating countries earmark in their R&D budgets the national contributions to Eurostars 
and finance their national participants. This funding mechanism is called a ‘virtual common 
pot’. In summary, Eurostars operates on the basis of centralised evaluation, but decentralised 
funding.   

The Eurostars Decision required an interim evaluation to be carried out by the Commission 
two  years  after  the  start  of  the  programme  and  for  the  results  to  be  communicated  to  the 
European  Parliament  and  the  Council.  The  interim  report  was  published  on  10  December 
20107 and the Commission submitted its report communicating these results in April 20118.  

The Eurostars Decision also required a final evaluation at the end of the programme, and for 
the results again to be communicated to the European Parliament and the Council. 

The  Makarow  group,  a  group  of  independent  experts  chaired  by  Marja  Makarow,  vice-
president of the Academy of Finland, carried out the final evaluation, which was published in 
November  2014.9  The  group  used  qualitative  and  quantitative  methods  to  assess  the 
relevance, efficiency, impact and sustainability of the programme. Compared with the interim 
evaluation, the numbers of applications, applicants and funded projects are about three times 
greater. In contrast to the interim evaluation, the Makarow group was therefore able to make 
quantitative calculations and carry out extensive econometric analyses. 

The  Makarow  group’s  main  conclusion  is  that:  ‘The  Eurostars  Joint  Programme  has 
succeeded  in  accelerating  the  growth  and  innovative  outputs  of  R&D-performing  SMEs. 
However,  several  aspects  of  governance  and  managerial  implementation  need  to  be 
improved.’ 

The main purpose of the current report is to communicate the results of the final evaluation to 
the European Parliament and the Council, as required by the Eurostars Decision. This report 
provides  a  short  summary  of  the  Commission’s  interim  evaluation  and  the  2011  report 
(section 2) followed by the Makarow group’s main findings and recommendations (section 3). 
In  line  with  the  2011  report,  the  Commission  also  submits  its  observations  on  the  main 
aspects  of  the  final  evaluation  (section  4).  Based  on  the  Makarow  group’s  report,  these 
observations are focused on those aspects of Eurostars that the Commission considers can be 
improved, in particular those that were already raised in the interim evaluation. 

The  Commission  also  takes  account  of  the  Eurostars-2  Decision  and  the  fact  that  several 
recommendations from both the interim and final evaluation have already been taken on board 
in this Decision and in the new delegation agreement. 

7   http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidence-

base/other_fp7_panel_evaluations/eurostars_programme_interim_evaluation.pdf#view=fit&pagemode=none. 

8   COM(2011) 186 final Brussels, 8.4.2011: 

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011DC0186&from=EN. 

9   https://ec.europa.eu/research/sme-techweb/pdf/ejp_final_report_2014.pdf. 

3 

 
                                                            
 
2. 

INTERIM EVALUATION AND THE COMMISSION'S 2011 REPORT 

Article 13(2) of the Eurostars Decision provides that: ‘An interim evaluation of the Eurostars 
Joint  Programme  shall  be  carried  out  by  the  Commission  two  years  after  the  start  of  the 
Eurostars Joint Programme and shall cover progress towards the objectives set out in Annex 
I. The evaluation shall also include recommendations on the most appropriate ways to further 
enhance  scientific,  management  and  financial  integration  and  assess  the  ability,  of  R&D-
performing SMEs in particular, to access the Eurostars Joint Programme and the quality and 
efficiency of its implementation. The Commission shall communicate the conclusions thereof, 
accompanied by its observations and, where appropriate, proposals for the amendment of this 
Decision, to the European Parliament and the Council.’      

The  2009  delegation  agreement  between  the  European  Commission  and  the  EUREKA 
Secretariat set out guidance on applying the  criteria of the Eurostars Decision and specified  
explicit questions to be addressed in the evaluation.10  

A  group  of  independent  experts11  appointed  by  the  Commission  submitted  the  interim 
evaluation  report  in  December  2010.  The  group  concluded  that:  ‘Eurostars  is  a  good 
programme, which meets its objectives and adds value to European R&D-performing SMEs. 
For this reason, the group of independent experts believes that Eurostars should not only be 
sustained  but,  preferably,  its  budget  should  be  increased  in  the  future.  However,  in  spite  of 
good progress, some scope for further improvement also remains.’ 

The interim evaluation took place in 2010 and the report contains data from the first four cut-
off  dates:  there  were  two  cut-offs  in  2008,  one  in  2009  and  one  in  201012.  A  total  of  1127 
applications from 3790 applicants had been received for the first four cut-offs. For the funding 
stage,  the  report  only  contains  data  from  the  first  three  cut-offs.  Funding  was  approved  for 
264  projects  with  total  projects'  cost  of  EUR  386  million.  Only  14  projects  had  been 
completed. In the two years since its inception, up to spring 2010, Eurostars mobilised a total 
earmarked budget of public funding of EUR 231.1 million, of which EUR 173.5 million came 
from the participating countries and EUR 57.6 million from the EU. 

For  the  first  two  cut-offs,  the  average  time-to-contract13  was  11.4  months.  This  time  varied 
significantly, from a minimum of 5.3 months to a maximum of 26.8 months in the first cut-off 
period  (average  11.8  months),  and  from  a  minimum  of  6.3  months  to  a  maximum  of  17.2 
months in the second (average 11.0 months). The annex to the delegation agreement (point 5, 

10   Eurostars  Joint  Programme  delegation  agreement  30-CE-0270684/00-14  (signed  in  June  2009),  Annex  1, 
point 4, pages 11 and 12. The Eurostars Decision (Article 4) states that the arrangements for the EU financial 
contribution  are  to  be  established  by  means  of  a  general  agreement  between  the  Commission  and  the 
EUREKA Secretariat. 

11   The  group  was  chaired  by  Ms  Anne  Laperrouze  (former  MEP).  The  other  members  of  the  group  were: 
Professor  Erkko  Autio  (Rapporteur),  Professor  Maja  Bucar,  Georg  Licht,  Professor  Jose  Molero  and 
Professor Lena Tsipouri. 

12   Eurostars is open to funding applications on a continuous basis, but the Secretariat sets cut-off deadlines for 

submitting applications. The central evaluation process starts after each cut-off. 

13   Time-to-contract  is  defined  as  the  time  between  the  cut-off  date  and  the  date  of  signature  of  the  grant 

agreement. 

4 

 
                                                            
‘Expected  results  and  indicators’)  gives  a  target  for  time-to-contract  of  six  months,  to  be 
achieved by the mid-term of the programme. 

The group considered the three main strengths of the programme to be as follows: 

1)  with a large number of applications, demand for the programme has exceeded original 
expectations and it has mobilised predominantly R&D-performing SMEs with close-
to-market, bottom-up projects;  

2)  programme management has improved over time in organising the necessary logistics 
and infrastructure and the degree of satisfaction of national authorities and participants 
is increasing; and 

3)  central  evaluation  is  considered  as  best  practice  in  terms  of  clear  and  transparent 

organisation and timely accomplishment.     

However, the group pointed to five areas where there is some scope for improvement:  

1)  harmonising and synchronising national procedures;  

2)  attracting  more  applicants,  in  particular  SMEs  with  no  prior  experience  of 

international collaboration;  

3)  improving  the  geographical  balance  of  technical  experts,  and  the  balance  between 

technical and market expertise within the pool of experts;  

4)  increasing visibility in tandem with ensuring the ability to fund more projects; and 

5)  continuing efforts towards the organisation of a virtual common pot.       

In its interim report to the European Parliament and the Council, the Commission stated that 
the  group’s  evaluation  report  covered  all  the  aspects  mentioned  in  Article  13(2)  of  the 
Eurostars  Decision,  and  provided  additional  facts,  comments  and  recommendations.  The 
Commission  considered  the  whole  report  to  be  an  integral  part  of  the  interim  evaluation 
process  set  out  under  Article  13(2).  In  its  comments,  the  Commission  highlighted  only  the 
most relevant findings and recommendations on the way forward.        

The Commission’s observations on the interim evaluation — which are still relevant in light 
of  the  final  evaluation  —  concerned  in  particular  how  to  further  enhance  scientific, 
management  and  financial  integration,  and  the  levels  of  quality  and  efficiency  in 
implementing Eurostars. 

The  Commission  agreed  with  the  group’s  recommendations  with  a  view  to  further 
accelerating the integration process. The Commission invited Eurostars countries to take the 
through  greater  harmonisation  and 
into  account 
programme’s 
simplification  of  funding  eligibility  rules  and  alignment  of  financing  and  disbursement 
practices. The Commission stressed that the Eurostars eligibility criteria should be universally 

integration  objective 

5 

 
applied across all participating countries; no country should be allowed to impose eligibility 
conditions going beyond those of Eurostars. The Commission also agreed with the proposal to 
test a partial (10 % of total earmarked funds) ‘real’ common pot approach.       

Regarding the quality and efficiency of implementation, the Commission supported proposals 
to  improve  the  central  evaluation  process  to  ensure  impartiality,  quality  and  timeliness. 
Experts must have the necessary expertise, and better feedback from evaluators to applicants 
would ensure transparency. The Commission also agreed with recommendations to reduce the 
time-to-contract. 

The Council gave its views on the interim evaluation in its conclusions of 31 May 201114. The 
Council ‘WELCOMES the view of the Group that Eurostars is well aligned with the objectives 
of Europe 2020, complements well the opportunities offered to SMEs in FP7 for international 
cooperation  and  has  proven  to  be  attractive  to  the  target  group  by  reaching  successfully 
European  R&D-performing  SMEs’;  ‘WELCOMES  the  recommendation  of  the  Group  to 
continue  Eurostars  beyond  2013;  and  WILL  CONSIDER  its  continuation  in  the  overall 
context of the future Common Strategic Framework for Research and Innovation funding’15.             

3. 

FINAL EVALUATION 

3.1. 

Introduction 

Article  13(3)  of  the  Eurostars  Decision  states  that:  ‘At  the  end  of  the  Eurostars  Joint 
Programme, the Commission shall conduct a final evaluation of the Programme. The results 
of the final evaluation shall be presented to the European Parliament and the Council.’ 

The Commission drafted terms of reference setting out the mandate, scope and objectives of 
the final evaluation. Together with the Eurostars Decision and the delegation agreement, the 
terms  of  reference  provided  the  basis  for  the  Makarow  group  regarding  approaches  to  be 
taken and questions to be addressed.    

The data presented by the Makarow group covers all 10 cut-off dates from 2008 to 2013. The 
EUREKA  Secretariat  received  3548  applications  comprising  11733  applicants.  The 
overwhelming majority of applicants were SMEs (72 %) and the average consortium size was 
3.3 participants. Funding was approved for 783 applications for a total of EUR 1 130 million. 
The  number  of  applications  continued  to  increase  throughout  the  duration  of  the  Eurostars 
programme.  However,  the  number  of  applications  approved  for  funding  did  not  rise 

14   The Council conclusions on the interim evaluation of Eurostars (Brussels, 1 June 2011, RECH 144, 

COMPET 226, MI 291): 
http://register.consilium.europa.eu/doc/srv?l=EN&f=ST%2011030 %202011 %20INIT    . 

15  In  the  Regulation  establishing  Horizon  2020,  the  European  Parliament  and  Council  decided  to  support 
research-intensive SMEs through the Eurostars programme. See Annex I, point 3(3)(b), p. 147, of Regulation 
(EU)  No  1291/2013  of  the  European  Parliament  and  of  the  Council  of  11  December  2013  establishing 
Horizon  2020  —  the  Framework  Programme  for  Research  and  Innovation  (2014-2020)  and  repealing 
Decision  No  1982/2006/EC 
(OJ  L  347,  20.12.2013,  p.  104):  http://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=OJ:L:2013:347:FULL&from=EN. 

6 

 
                                                            
accordingly,  meaning  that  the  success  rate16  declined  from  42 %  in  2008  to  17 %  in  2013. 
Total estimated public funding for 2008-2013 was EUR 472 million, giving a proportion of 
EU funding (EUR 100 million) to national funding (EUR 372 million) of 26.9 % (or 21.2 % 
of the total public funding. 

The  average  time-to-contract  has  improved  compared  with  the  situation  at  the  time  of  the 
interim  evaluation.  For  example,  in  the  third  cut-off  period  (2009),  the  average  time-to-
contract  was  435  days,  whereas  by  the  eighth  cut-off  in  2012,  it  had  fallen  to  282  days. 
However,  the  time-to-contract  is  still  fairly  long  in  a  significant  number  of  countries.  The 
Makarow group considers that the programme’s time-to-contract must be improved urgently, 
as the existing considerable differences between participating countries calls the whole system 
into question.    

3.2.  Methodologies 

The Makarow group used qualitative and quantitative tools for the final evaluation. The main 
methods  were:  desk  research;  interviews  with  stakeholders  and  SMEs;  an  online  survey  of 
SMEs that had applied for Eurostars; and quantitative analyses. 

Through  desk  research,  the  group  gathered  documents  from  various  stakeholder  groups  and 
analysed  the  data  and  information  relevant  for  the  Eurostars  evaluation  criteria.  This  desk 
research  enabled  the  group  to  identify  major  issues,  challenges  and  questions  on  the 
implementation  of  Eurostars.  The  source  material  included  legislative  and  cross-cutting 
documents,  minutes  of  the  EUREKA  high-level  group  meetings,  statistical  data  and  data 
hosted by the EUREKA Secretariat. 

The Makarow group held interviews in Brussels with two representatives of the Commission, 
three  representatives  of  EUREKA  and  with  the  EUREKA  Secretariat’s  top  management. 
Individual  group  members  conducted  face-to-face  or  telephone  interviews  with  20 
representatives  of  national  governing  and  administrative  bodies  of  Eurostars  and  with  the 
chief executive officers of 26 SMEs, operating in 12 different countries. The group drew up 
guidelines to structure the interviews, the guidelines being adjusted according to the function 
of  the  interviewees.  All  interviews  were  documented  in  writing.  The  interviews  covered  a 
wide range of topics. 

As  part  of  the  evaluation,  an  online  survey  asked  participants  about  their  experience  with 
Eurostars. The target group of the online survey was restricted to R&D-performing SMEs and 
other SMEs. The online survey ran from 9 May to 10 July 2014. The survey reached out to 
6 620 SMEs through emails linking to an online questionnaire. Two different questionnaires 
were prepared: the first targeted funded projects, the second targeted non-funded projects. A 
total  response  rate  of  46 %  was  obtained  (72 %  for  funded  and  39 %  for  the  non-funded 
SMEs).       

16  The  success  rate  is  defined  as  the  total  number  of  applications  approved  for  funding  divided  by  the  total 

number of applications. 

7 

 
                                                            
To  understand  the  extent  of  the  Eurostars  impact,  the  Makarow  group  developed  an 
econometric  counterfactual  impact  evaluation.  The  econometric  study  focused  primarily  on 
job  creation  in  participating  SMEs  and  on  innovation  output  as  measured  by  patent 
applications.  With  respect  to  data  gathering,  this  required  that  the  EUREKA  Secretariat’s 
SME  applicant  data  be  matched  to  the  Bureau  van  Dijk’s  Amadeus  database,  from  which 
information  on  employment  figures  were  gathered.  In  addition,  the  EUREKA  Secretariat's 
SME  applicant  data  were  matched  with  the  European  Patent  Office  Worldwide  Patent 
Statistical  Database.  The  resulting  database  brings  together  applicant  information  and  time 
series  data  on  employment  and  patent  applications,  enabling  estimations  of  econometric 
treatment effects to be made. 

3.3. 

Findings and assessment 

•  Target group and scope of the programme 

The Makarow group points out that, according to participating SMEs and the governing and 
administrative bodies of Eurostars, the programme is relevant to the target group’s needs and 
objectives. The high level of demand from SMEs is considered an indication of this ‘strategic 
fit’.  Almost  all  participating  SMEs  plan  to  apply  to  Eurostars  again,  as  the  programme 
corresponds 
the 
overwhelming majority of participants in Eurostars, and received about 75 % of the funding. 
In addition, the programme was successful in addressing young, small and micro firms. 

their  needs  and  objectives.  R&D-performing  SMEs  count  for 

to 

in  national  and 

Participating  SMEs  already  have  experience 
international  funding 
programmes,  and  the  majority  already  have  pre-existing  international  links.  However,  the 
Makarow group found that applications involving university or public research organisations 
have a higher  chance  of  being funded.  This may be because the presence of a university or 
public  research  organisation  signals  the  high  quality  of  the  proposal,  or  that  those 
organisations  are  more  experienced  in  preparing  application  documents,  or  that  those 
applications provided the best balance between scientific excellence and market knowledge. 

Based on its bottom-up nature, Eurostars is, according to the group, able to address a wide-
variety of innovative technologies, ranging from information and communication technology 
and  its  applications,  and  medical  biotechnology,  to  (renewable)  energy  and  steering  and 
control technologies. 

•  Governance 

The  Makarow  group  notes  that  the  EUREKA  Secretariat  is  generally  acknowledged  as  a 
successful  dedicated  implementation  structure.  However,  there  is  a  need  to  improve 
managerial  and  administrative  practices,  enhance  the  qualifications  of  staff,  develop  and 
implement  a  marketing  and  promotion  strategy,  and  develop  and  maintain  a  well-structured 
information  management  system  and  database  for  use  by  the  Secretariat  and  the  other 
Eurostars governing and administrative bodies. 

8 

 
The  group  considers  that  the  Eurostars  governance  system  functions  properly  and  that 
interaction  between  national  and  central  administrative  bodies  is  effective.  However,  the 
international dimension of Eurostars required a complex governance system to be set up. 

The  EUREKA  high-level  group  regularly  discusses  various  issues  critical  to  the  success  of 
Eurostars without necessarily reaching a consensus for action or agreeing binding rules for all 
countries. Synchronisation and harmonisation of funding, eligibility rules and the adequacy of 
national budgets have been intensely analysed and discussed. The Makarow group considers 
that  there  is  a  need  to  increase  the  high-level  group’s  ownership  of  and  involvement  in 
Eurostars.  This  is  critical  to  facilitate  synchronisation  and  harmonisation.  Similarly,  the 
national funding bodies’ involvement remains limited in the governance structure; they should 
be more involved in order to achieve greater synchronisation and harmonisation. 

Finally, Eurostars has strengthened the SME presence within EUREKA and contributed to a 
much  higher  visibility  of  their  contribution  to  European  competitiveness  and  innovation.  In 
this  way,  Eurostars  has  had  a  significant  and  long-lasting  positive  effect  on  the  EUREKA 
network. 

•  Management and operations 

The  Makarow  group  considers  that  the  virtual  common  pot  has  worked  well  and  is  the 
preferred funding mechanism, since it attracts countries to contribute to Eurostars. In response 
to the programme’s success, several countries increased their contribution rates significantly 
beyond the initially agreed level. However, the full benefits of the virtual common pot have 
not yet been exploited in Eurostars, since some countries, in particular Germany and Spain, do 
not  allocate  a  sufficient  level  of  funding  for  the  programme.  Because  of  this,  some  good 
projects selected fall from the ranking list, which in turn leads to frustration among applicant 
SMEs. 

The group reports that synchronisation of funding is still the most important challenge facing 
Eurostars,  although  some  improvements  have  been  achieved.  In  some  cases,  participating 
SMEs  face  great  difficulties  because  of  the  longer  time-to-contract.  There  are  also  parallel 
application  and  evaluation  requirements  and  dual  reporting  obligations  applied  by  some 
national funding bodies. 

The other major challenge facing Eurostars is the harmonisation of funding rules. The high-
level group considers that this challenge should be addressed through a top-down process at 
the  initiative  of  national  ministries  and  the  EU. National  project  coordinators  think  that  full 
harmonisation is not possible: for example, they cannot change the funding percentage or the 
financing  upper  limit  for  each  SME.  Another  issue  is  the  need  for  greater  harmonisation  of 
subcontracting rules for universities and public research organisations. 

The  Eurostars  central  evaluation  process  has  worked  well.  However,  the  Makarow  group 
considers it necessary to improve the transparency and feedback mechanism and to streamline 
the process in order to decrease the time from the initial application to the availability of an 

9 

 
evaluation  report.  The  time-to-contract  was  rather  long  at  the  start  of  the  Eurostars 
programme, but decreased considerably during the first phase. 

Although the number of technical experts and of those with market experience has increased 
considerably  since  2012,  there  is  still  room  to  improve  geographical  and  gender  balance. 
National project coordinators would also like to be involved in the process of improving the 
experts’  database.  SMEs  are  satisfied  with  the  level  of  information  they  receive  about  the 
evaluation of their projects. They deem the reporting workload and the monitoring process to 
be appropriate, and they also find the evaluation process clear, transparent and timely. 

•  Funding 

The  Makarow  group  highlights  that  one  indicator  of  the  success  of  Eurostars  was  the 
significant increase in the number of applications and hence the increasing demand for public 
funding. The annual number of applications submitted increased from 215 in 2008 to 948 in 
2013. During the lifetime of Eurostars, however, this resulted in a decline in the proportion of 
submitted  applications  that  were  funded.  The  success  rate  fell  from  42 %  in  relation  to  the 
first  cut-off  date  to  17 %  for  the  tenth.  The  proportion  of  projects  meeting  the  quality 
threshold that were approved for funding also declined, from 68 % for cut-off 1 to 55 % for 
cut-off  10;  in  short,  an  increasing  proportion  of  high-quality  projects  were  not  funded.  The 
Makarow  group  considers  that  the  increase  in  the  number  of  applications  per  year  is 
‘remarkable’ and ‘impressive’. 

National  budgets  for  Eurostars  still  vary  enormously.  This  reflects  the  relative  size  of  the 
country  and  the  organisation  of  the  national  public  funding  system,  and  also  demand  from 
national SMEs for Eurostars funding. These differences had a significant negative impact on 
the  functioning  of  the  virtual  common  pot  system.  The  size  of  funding  per  participant  is 
judged as adequate by most participants and programme administrators. 

The  Makarow  group  considers  that  Europe’s  financial  and  economic  crisis  had  a  negative 
effect on national funding for Eurostars in the majority of countries. The number of projects 
above  threshold  decreased  considerably  owing  to  lack  of  funds,  and  for  this  reason  some 
countries did not participate in the latest calls for tender. The countries that were not short of 
funds also reported that they were affected negatively by the crisis, since the SMEs that they 
supported were partnered with counterparts in countries that were having funding difficulties. 

•  Uptake of recommendations from the interim evaluation report 

The  Makarow  group  assessed  the  uptake  of  recommendations  from  the  interim  evaluation 
report.  Of  21  policy  recommendations  to  be  implemented  by  2013,  the  Makarow  group 
considers that seven recommendations have been fully implemented, ten have begun to be — 
or been partially — implemented, while four have not been implemented at all. 

10 

 
These four recommendations concern:  

1)  setting a global deadline for the signature of grant agreements;  

2)  giving the EUREKA high-level group an active role in monitoring the time-to-contract 

and in ensuring that countries that lag behind streamline their processes;  

3)  experimenting with the approach of allocating 10 % of total earmarked funds to a real 

common pot; and 

4)  having  the  EUREKA  Secretariat  initiating  an  activity  to  explore  the  integration  of 
EUREKA cluster initiatives in view of learning and implementing integration lessons 
that  could  be  applied  to  promote  the  integration  of  Eurostars  with  national  R&D 
support initiatives.         

3.4. 

Recommendations 

The Makarow group’s report contains 28 recommendations. 

•  Target group and scope of the programme 

1)  The  EUREKA  Secretariat  should  develop  an  action  plan  to  promote  Eurostars 
participants’ contacts with R&D and innovation intermediaries mostly involved in the 
project preparation phase.  

2)  The Secretariat should promote linkages between Eurostars participants and different 
types of EUREKA projects in order to facilitate Eurostars projects’ connections with 
clusters, 'umbrellas' and other thematic or individual projects. 

3)  The Secretariat should organise brokerage events, such as pitch competitions, between 
Eurostars  participants  and  financing  institutions  to  increase  the  possibilities  of 
successful  projects  to  receive  follow-up  funding  from  the  private  sector.  A 
dissemination  strategy  should  be  developed  to  target  financiers,  other  SMEs,  and 
national and international government bodies. 

4)  The Secretariat should raise awareness among the Eurostars beneficiary SMEs of the 

importance of protecting the intellectual property rights of their innovations. 

5)  The  Secretariat  and  national  project  coordinators  should  ensure  that  the  programme 
also reaches SMEs that do not have pre-existing international links, by designing and 
implementing a marketing and promotion strategy. 

6)  The  Secretariat  and  the  high-level  group  should  agree  on  the  definition  and 
development time of ‘near market R&D’ projects for Eurostars, which is neutral vis-à-
vis the specificities of particular technologies and markets. 

11 

 
7)  The  European  Commission  and  the  Secretariat  should  check  the  possibilities  for 
linking  Eurostars  and  the  new  Horizon  2020  SME  instrument  to  capitalise  on  the 
potential growth and competitive results of Eurostars SME participants. 

•  Governance 

1)  Decisions  taken  by  the  high-level  group  should  be  translated  without  delay  into 
tasks,  and  allocation  of 
implementation  plans  with  concrete  objectives, 
responsibilities, using a SMART (specific, measurable, attainable, relevant and time-
based) approach. 

2)  The  high-level  group  should  identify  countries  that  impose  parallel  application  and 
evaluation requirements as well as dual reporting obligations to Eurostars projects, and 
agree  with  them  to  eliminate  the  national  processes.  The  high-level  group  should 
ensure that this is implemented before the spring cut-off of Eurostars-2 in 2016. 

3)  The Secretariat should improve the transparency of the central evaluation process and 
feedback  mechanisms  to  applicants,  national  funding  bodies  and  national  project 
coordinators.  The  coordinators  should  have  access  to  the  consensus  reports  of  the 
independent  evaluation  panel,  as  well  as  to  the  individual  reports  of  the  technical 
evaluators, to be able to give detailed feedback to applicants. 

4)  The Secretariat should ensure a more balanced geographical distribution of technical 
experts  in  close  cooperation  with  national  project  coordinators.  Attention  should  be 
paid to achieve gender balance as well. In addition, the Secretariat should expand the 
pool of technical experts with appropriate knowledge of markets. 

•  Management and operations 

1)  The EUREKA Secretariat should maintain the frequency of two cut-offs per year. 

2)  The Secretariat should decrease the time from application (cut-off date) to evaluation 
to  4-5  months  so  that  results  of  the  central  evaluation  will  be  available  to  national 
project coordinators and applicants within five months (150 days) at the latest. Every 
national  funding  body  needs  to  sign  the  funding  contracts  within  2-3  months.  It  is 
recommended that the EUREKA Secretariat, together with the chair of the high-level 
group and the heads of national funding bodies, reviews the programme management 
processes  of  those  national  funding  bodies  that  risk  not  meeting  the  deadline,  and 
agrees with them the steps to be taken to shorten the process. 

3)  A standard consortium agreement should be provided to applicants in order to speed 

up the contractual demands of project participants. 

4)  The  Secretariat  should  establish  and  maintain  a  robust  central  Eurostars  database  to 
collect,  store  and  process  reliable  data  on  projects  and  beneficiaries.  It  should 
introduce  quality  checks  of  data  for  the  application  forms  as  well  as  for  project 
progress,  final  and  market  impact  report  forms.  The  deadline  for  submission  of  the 

12 

 
final  reports  should  be  reduced  to  one  year  and  market  impact  reports  should  be 
available within two years of the end of the project. 

5)  The Secretariat should modify the application form and the project progress, final and 
market  impact  report  forms.  Detailed  revisions  of  the  current  specific  questions  and 
headings  in  the  forms,  as  well  as  re-ordering,  deletion  and  addition  of  a  number  of 
questions,  are  required  to  improve  the  logic  of  the  forms.  Correspondingly,  the 
guidelines for these forms should be adjusted accordingly. 

6)  The  Secretariat  should  develop  a  dedicated  application  form  and  corresponding 
guidelines  for  resubmissions.  The  resubmissions  need  to  include  an  explanation  on 
what has been changed, especially in response to comments from previous evaluations 
received via the national project coordinators. 

7)  The  Secretariat  should  design  and  implement  an  institutional  development  plan  to 
strengthen its programme and organisational management practices in order to be able 
to  improve  the  implementation  of  Eurostars  in  line  with  international  best  practices, 
and to hire, retain and motivate qualified staff experienced in the management of R&D 
and innovation programmes. It is recommended that the Secretariat receives mentoring 
from the network’s national funding bodies with relevant competencies. 

8)  The Secretariat should introduce a unique identifier for all organisations, not only for 
SMEs  and  large  firms  but  also  for  universities,  public  research  organisations,  and 
other  partners  of  R&D-performing  SMEs.  If  available,  each  organisation  should 
provide  their  Participant  Identification  Code  under  Horizon  2020.  The  application 
form should include information about whether the project was submitted previously 
to another national or EU programme. 

9)  The  Secretariat  should  collect  the  information  from  all  countries  about  the  time-to-
moneyand  submit  this  information  to  the  high-level  group,  and  agree  on  a  binding 
deadline for collection and submission. 

10) Information on the exact start and end dates of the projects needs to be included in the 
central Eurostars database. National project coordinators should without delay inform 
the EUREKA Secretariat on signed grant agreements, including start and end dates of 
the projects. 

11) The  Secretariat,  the  high-level  group  and  national  funding  bodies  should  ensure  that 
the time-to-contract will not exceed 7 months (210 days) as agreed for Eurostars-2 in 
the Budapest document. 

12) From  the  spring  cut-off  of  2016  onwards,  the  Secretariat,  the  high-level  group  and 
national  funding  bodies  should  agree  that,  for  at  least  half  of  the  projects,  the  grant 
agreements between national funding body and programme participants will be signed 
within  210  days  of  the  cut-off  date.  For  90 %  of  the  projects,  all  grant  agreements 
should be signed within one year from the cut-off date.   

13 

 
13) The  Secretariat  and  national  project  coordinators  should  jointly  define  and  develop 
tools  to  facilitate  information  exchange  between  them,  improve  the  quality  and 
quantity  of  information  available  from  and  for  both  sides,  allow  monitoring  of  all 
project phases, and reduce the administrative cost of bilateral information exchanges. 

14) The  European  Commission  should  consider  not  granting  funding  to  those  Eurostars 
projects  for  which  all  grant  agreements  have  not  been  signed  within  one  year  of  the 
cut-off deadline unless the delay in signing a funding contract is caused by members 
of the consortium funded. 

•  Funding 

1)  The  high-level  group  is  strongly  encouraged  to  agree  on  a  minimum  baseline  for 
funding rules to be harmonised and to ensure that each participating country conforms 
to this. The maximum funding rates should be the same for each type of partner in all 
countries. The process should be closely followed up by the high-level group and be 
completed by the first cut-off in 2016. 

2)  The funding of universities and public research organisations should be harmonised so 
that they participate as partners of the foreign R&D-performing SMEs and not only as 
subcontractors of R&D-performing SMEs from the same country. 

4. 

COMMISSION’S OBSERVATIONS 

4.1. 

Introduction 

The Commission considers that the Makarow group report has addressed the main aspects that 
were to be covered by the evaluation, as described in the Eurostars Decision, the delegation 
agreement and in the terms of reference.   

As Member States and other participating countries have increased their funding contribution 
to Eurostars — estimated at EUR 372 million — the EU contribution of EUR 100 million, at 
26.9 %, fulfils the requirement of the Eurostars Decision that EU funding should not exceed 
one third of the contributions of Member States and other participating countries.  

The  Eurostars  Decision  refers  to  the  importance  of  SMEs  for  European  growth  and 
competitiveness. The high and increasing number of applications for funding from Eurostars, 
and the results obtained by the supported SMEs, indicates that the programme is relevant for 
the growth of R&D-performing SMEs. 

Since the start of the programme in 2008, several aspects of Eurostars improved up to the end 
of 2013. For example, time-to-contract was shortened and harmonisation and synchronisation 
improved. The Commission welcomes this and appreciates the efforts made by the EUREKA 
Secretariat and national funding bodies.         

The  Makarow  group’s  main  conclusion  is  that  Eurostars  has  succeeded  in  accelerating  the 
growth and innovative outputs of R&D-performing SMEs. However, the group also considers 
that aspects of governance and managerial implementation need to be improved. 

14 

 
The  first  part  of  the  main  conclusion  is  substantiated  by  extensive  econometric  impact 
analysis. This shows that the employment growth rate of R&D-performing SMEs funded by 
Eurostars  was  nearly  twice  as  high  as  that  of  applicant  SMEs  that  were  not  funded. 
Concerning  the  second  part,  the  group  concluded  that  the  lack  of  harmonised  funding  rules 
and  synchronised  national  processes  led  to  inefficiencies.  The  group  also  considers  that  the 
effectiveness of central governance, administration and operations needs to be improved.       

The  Commission  finds  that  the  findings,  assessment  and  recommendations  of  the  final 
evaluation  have  been  convincingly  substantiated.  The  Commission  agrees  with  the  main 
conclusion of the Makarow group. 

Several  of  the  recommendations  from  the  interim  evaluation  and  the  final  evaluation  have 
already been taken on board in the Eurostars-2 Decision (2014-2020).17   

As  the  Makarow  group’s  report  constitutes  an  integral  part  of  the  evaluation  process,  the 
Commission does not comment in the following observations on every detail of the report. As 
the  Commission’s  focus  is  forward-looking,  these  observations  are  mainly  concerned  with 
aspects  of  Eurostars  for  which  the  Commission  considers  there  is  the  greatest  scope  for 
improvement, i.e. the second part of the Makarow group's main conclusion.   

4.2. 

Target group and scope of the programme 

Both  the  interim  evaluation  and  the  Makarow  group  report  indicate  that  the  EUREKA 
Secretariat and national project coordinators should ensure that Eurostars also reaches SMEs 
that  do  not  have  pre-existing  international  links.  The  Commission  agreed  with  this 
recommendation  in  2011,  and  the  Commission  still  agrees  in  2015.  Although  this 
recommendation  is  addressed  to  both  the  Secretariat  and  the  coordinators,  several  other 
recommendations concerning the promotion of — or raising awareness about — Eurostars are 
addressed only to the Secretariat (cf. recommendations 3, 4, 5 and 7 above). The Commission 
agrees  with  these  recommendations.  However,  although  the  EUREKA  Secretariat  has  a 
pivotal  role,  participating  Eurostars  countries  and  national  funding  bodies  also  have  a 
responsibility  for  reaching  more  and  better  R&D-performing  SMEs  with  a  potential  for 
growth. 

The Makarow group recommends that the Commission and the Secretariat should check the 
possibilities  for  linking  Eurostars  and  the  new  Horizon  2020  SME  instrument  (cf. 
recommendation  6).  Although  it  is  unclear  what  ‘linking’  means,  the  Commission  analysed 
the  strategic  positioning  of  the  different  instruments  when  the  proposal  for  Eurostars-2  was 
presented.18  The  Commission  considers  that  the  support  they  provide  occurs  at  different 
phases  in  the  development  of  the  SMEs  research  and  business  ideas  with  relation  to  the 
market.  Eurostars  is  different  and  complementary  —  not  supplementary  —  to  the  SME 
instrument in Horizon 2020. The delegation agreement also states that national funding bodies 

17 For example, the Eurostars-2 Decision states that: ‘As part of the improvements from the previous Eurostars 
programme,  Eurostars-2  should  head  towards  shorter  time-to-grant,  stronger  integration  and  lean, 
transparent and more efficient administration to the ultimate benefit of research and development-performing 
SMEs.’ 

18 Brussels, 10.7.2013, SWD(2013) 242 final. 

15 

 
                                                            
should  take  effective  measures  to  avoid  any  double  funding  of  final  beneficiaries’  projects 
through  other  EU  funding  sources.  That  Eurostars  and  the  SME  instrument  have  different 
target  groups  needs  to  be  clearly  communicated  by  the  EUREKA  Secretariat,  the  European 
Commission and the Executive Agency for Small and Medium-sized Enterprises.     

4.3. 

Governance 

The  Makarow  group  notes  that  the  EUREKA  high-level  group  regularly  discusses  various 
issues critical to the success of Eurostars without necessarily reaching a consensus for action 
or  agreeing  binding  rules  for  all  countries.  The  group  considers  it  necessary  to  increase  the 
high-level group’s ownership of and involvement in Eurostars. 

The Commission agrees with this assessment and the recommendation that decisions by the 
high-level group should be taken under a SMART (specific, measurable, attainable, relevant 
and time-based) approach (recommendation 8). One concrete example of such an approach is 
recommendation 9 for the elimination of parallel application and evaluation requirements and 
dual  reporting  before  the  first  cut-off  date  in  2016.  The  Commission  fully  supports  this 
recommendation. 

The  Makarow  group  also  strongly  encourages  the  EUREKA  high-level  group  to  agree  on  a 
minimum baseline upon which to harmonise funding rules (recommendation 27). The group 
considers  that  this  should  be  completed  by  the  first  cut-off  date  in  2016.  The  Commission 
considers  that  improvements  still  remain  to  be  made  regarding  harmonisation  of  national 
funding  rates  and  funding  rules.  The  Commission  therefore  fully  agrees  with  this 
recommendation.     

4.4.  Management and operations 

Time-to-contract — and the differences in time-to-contract across participating countries  — 
was considered in the interim evaluation and has also been analysed by the Makarow group. 

Time-to-contract  is  determined  at  two  levels.  First,  the  central  evaluation  is  carried  out  and 
coordinated  by  the  EUREKA  Secretariat.  Second,  the  funding  allocation  and  contract 
negotiations take place at national level. Both the interim evaluation and the Makarow group 
found that the quality of the central evaluation is quite good and runs smoothly. 

However, the Makarow group recommends several concrete measures at both levels to reduce 
time-to-contract  (cf.  recommendations  14,  15  and  16  above).  The  Commission  agrees  with 
these  recommendations,  but  it  seems  clear  that  the  main  problem  is  the  huge  differences  at 
national level. The Commission considers that the EUREKA high-level group and Secretariat, 
and the national funding bodies, should give priority to reducing these differences. 

The  Makarow  group  also  recommends  that  the  Commission  should  consider  withholding 
grant funding for any Eurostars project for which all grant agreements have not been signed 
within one year of the cut-off deadline (recommendation 17 above). 

The Budapest document stated that, for Eurostars-2, national funding bodies ‘shall be aiming 
at a time-to-contract of 7 months after the cut-off deadline, but preferably faster’. The 2014 

16 

 
Eurostars-2  Decision  set  an  objective  to  increase  efficiency  of  public  funding  ‘by  aligning, 
harmonising  and  synchronising  the  national  funding  mechanisms  of  participating  states’ 
(Article  3).  The  Decision  also  clearly  states  that  the  Union’s  financial  contribution  is 
conditional  upon  demonstration  by  participating  states  that  they  have  set  up  Eurostars-2  in 
accordance  with  the  objectives  laid  down  in  Article  3.  The  same  provisions  are  also  to  be 
included  in  bilateral  agreements  signed  between  the  EUREKA  Secretariat  and  national 
funding  bodies.  Such  agreements  ‘shall  include  the  rules  governing  the  transfer  of  the 
Union’s  contribution  and  the  minimum  operational  targets  and  national  progressive 
milestones  for  further  integration  and  synchronisation  of  national  programmes,  including  a 
shorter  time-to-grant  in  accordance  with  Regulation  (EU,  Euratom)  No  996/2012  and 
Regulation  (EU)  No  1290/2013’  (Annex  I,  paragraph  11).  Horizon  2020  also  contains  clear 
rules regarding time-to-grant: applicants are to be informed within five months of the outcome 
of  the  scientific  evaluation  of  their  applications,  and  grant  agreements  are  to  be  signed  or 
grant decisions notified within three months from the date of informing applicants they have 
been  successful.19  The  Commission 
the  Makarow  group’s 
recommendation  17  and  calls  on  participating  states  to  respect  their  own  commitment  to 
reduce the time-to-contract.     

therefore  fully  supports 

The Makarow group also made several recommendations regarding the EUREKA Secretariat 
database, including how data is collected and entered into the database (cf. recommendations 
19 — 24 above). 

The  Commission  agrees  that  the  EUREKA  Secretariat  application  process  and  database 
should  be  improved.  The  forms  to  be  filled  in  by  applicants  must  be  as  logical  and  easy  to 
understand as possible. Logic checks for fields requiring arithmetic completion should be in 
place  to  reduce  the  probability  of  entering  wrong  values.  A  uniform  approach  regarding 
terminology  should  be  introduced,  both  in  the  forms  and  in  the  guidance  to  users.  The 
technological  area  classification  should  also  follow  the  nomenclature  of  the  EU’s  statistical 
classification  of  economic  activities  (NACE  Rev.2)  in  order  to  provide  a  common  basis  for 
the comparative evaluation of different EU funding schemes.     

4.5. 

Role of universities and public research organisations 

The  Makarow  group  recommends  that  the  funding  of  universities  and  public  research 
organisations be harmonised so that they participate as partners and not only as subcontractors 
(cf. recommendation 27). 

The  Commission  understands  this  recommendation  to  mean  that  a  project  or  consortium 
could, for example, consist of an SME in one country, a university in a second country and a 
public  research  organisation  in  a  third  country.  The  Commission  considers  that  this  would 

19See  Article  20  of  Regulation  (EU)  No  1290/2013  of  the  European  Parliament  and  of  the  Council  of  11 
December 2013 laying down the rules for participation and dissemination in ‘Horizon 2020 — the Framework 
Programme  for  Research  and  Innovation  (2014-2020)’  and  repealing  Regulation  (EC)  No  1906/2006  (OJ  L 
347, 
http://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=OJ:L:2013:347:FULL&from=EN. 

20.12.2013, 

81): 

p. 

17 

 
                                                            
probably  lead  to  universities  and  public  research  organisations  playing  a  bigger  role  in 
Eurostars projects and that a bigger part of the budget would go to such institutions.    

The Eurostars Decision provides that the main activity in Eurostars consists of R&D-activities 
led  by  one  or  more  R&D-performing  SMEs.  Although  universities,  public  research 
organisations and large companies may also participate in the programme, the target group for 
Eurostars is clearly R&D-performing SMEs. 

However,  Horizon  2020  dedicates  substantial 
research 
collaborations  between  actors  belonging  to  the  'Knowledge  Triangle'  of  research,  business, 
and higher education, through the Marie Skłodowska-Curie actions and the European Institute 
of Innovation & Technology. 

intra-European 

funding 

to 

The Commission therefore does not support this recommendation.        

5.  Conclusion 

The  Eurostars  programme  has  shown  good  results  in  terms  of  accelerating  the  growth  and 
innovative  outputs  of  R&D-performing  SMEs.  The  final  evaluation  provides  evidence  that 
there is a need for Eurostars and that the programme is reaching its target group. 

However, the Commission agrees with the Makarow group that the most important challenge 
for  Eurostars  is  to  synchronise  funding  and  to  harmonise  funding  rules.  Although  some 
progress  has  been  made,  it  is  essential  therefore,  for  the  long-term  effectiveness  of  the 
programme,  that  the  EUREKA  high-level  group  and  Secretariat,  and  the  national  funding 
bodies,  make  increased  efforts  to  accelerate  synchronisation  and  harmonisation  under  the 
Eurostars programme. 

18