COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Ex-post Evaluation of the 2nd Health Programme 2008-2013 Decision No 1350/2007/EC establishing a second programme of Community action in the field of health (2008-13)
Date: 2016-05-10
Institutions: Directorate-General for Health and Food Safety, European Commission
Subjects: EU Member State, EU financing, EU programme, disease prevention, health care, health care system, project evaluation, public awareness campaign, public health

EN 

   EN 

   EUROPEAN COMMISSION  Brussels, 10.5.2016  SWD(2016) 148 final   COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS  Ex-post Evaluation of the 2nd Health Programme 2008-2013  Decision No 1350/2007/EC establishing a second programme of Community action in the field of health (2008-13)  {COM(2016) 243 final} {SWD(2016) 149 final}  
 
 
Table of contents 

1. INTRODUCTION .................................................................................................................................... 2 

1.1. Purpose of the evaluation ......................................................................................................................... 2 

1.2. Scope of the evaluation ............................................................................................................................ 2 

2. BACKGROUND TO THE INITIATIVE ............................................................................................... 2 

2.1. Description of the initiative and its objectives ......................................................................................... 2 

2.2. What outputs were expected from the Programme? ................................................................................ 3 

2.3. What results and impacts were expected from the Programme? ............................................................. 5 

2.4. Baseline ................................................................................................................................................... 5 

3. EVALUATION QUESTIONS ................................................................................................................. 6 

4. METHOD .................................................................................................................................................. 8 

4.1. Process and methods used........................................................................................................................ 8 

4.2. Limitations – robustness of findings ........................................................................................................ 9 

4.3. Quality assessment of the study ............................................................................................................. 10 

5. IMPLEMENTATION OF THE PROGRAMME ................................................................................ 11 

6.1.   Budget distribution per financial mechanism ...................................................................................... 11 

6.2.   Budget distribution per thematic area ................................................................................................. 12 

6.3.   Participation by type of stakeholder group .......................................................................................... 14 

6.4.   Participation by geographical area ...................................................................................................... 14 

6. FINDINGS OF THE EVALUATION ................................................................................................... 16 

7.1.   Programme management ..................................................................................................................... 16 

Effectiveness and efficiency of the Programme management ...................................................................... 18 

7.2.   The Programme's effectiveness and factors that are influencing it ..................................................... 20 

7.4.   Coherence and consistency with other European policies and programmes ....................................... 23 

7. CONCLUSIONS ..................................................................................................................................... 24 

ANNEX I – EVALUATION MATRIX ..................................................................................................... 26 

ANNEX II – RECOMMENDATIONS APPLIED ................................................................................... 33 

ANNEX III – JOINT ACTIONS ACHIEVEMENTS 2008 – 2013 ......................................................... 41 

ANNEX IV – QUALITY ASSESSMENT FORM .................................................................................... 46 

1 

 
 
 
1.  

INTRODUCTION 

1.1.   Purpose of the evaluation 

This staff working document accompanies the Commission report and the external evaluation report that 
the  Commission  is  transmitting  to  the  European  Parliament,  the  Council,  the  European  Economic  and 
the  Regions  under  Article  13(3)(c)  of  Decision 
Social  Committee  and 
No 1350/2007/EC.1 

the  Committee  of 

The  external  and  independent  ex  post  evaluation  of  the  2nd  Health  Programme  was  conducted  in  2014-
2015. Its purpose was primarily to assess the performance of the Programme management implementation, 
including  follow-up  to  the  recommendations  in  past  health  programme  evaluations.  The  evaluation 
contributes to a better understanding of the strengths and weaknesses of Programme implementation and 
management and provides conclusions that can be used as a basis for improving the implementation of the 
current 3rd Health Programme. 

1.2.   Scope of the evaluation 

This  evaluation  follows  on  from  previous  evaluations  of  the  1st  Public  Health  Programme  and  the  2nd 
Health Programme, building on the results, in particular of the mid-term evaluation of the latter, without 
repeating earlier evaluation work carried out to inform the design of the 3rd Health Programme.  

Consequently,  it  focuses  on  specific  aspects  of  the  Programme,  such  as  programme  management, 
dissemination  of  results  and  synergies  with  other  programmes,  seeking  to  complement  the  previous 
evaluations.  While  addressing  the  functioning  of  the  entire  Programme,  the  contractors  concentrated  on 
issues that were insufficiently explored in past exercises and provided conclusions that can form a basis for 
changes to improve the effectiveness and efficiency of the 3rd Health Programme. 

2.    BACKGROUND TO THE INITIATIVE 

2.1.   Description of the initiative and its objectives 

According  to  the  Treaty  on  the  Functioning  of  the  European  Union  (TFEU),  a  high  level  of  health 
protection  shall  be  ensured  in  the  definition  and  implementation  of  all  Union  policies  (Article  168  (1) 
TFEU).    Union  action,  which  shall  complement  national  policies,  shall  be  directed  towards  improving 
public  health,  preventing  physical  and  mental  illness  and  diseases,  and  obviating  sources  of  danger  to 
physical and mental health. 

The 2nd Health Programme was the main instrument for implementing the EU’s 2008-2013 health strategy 
Together for health;2 from 2011, it was aligned with the priorities of the Europe 2020 strategy.3  

The Programme’s overall aim was to complement, support and add value to Member States’ policies 
and  to  contribute  to  increased  solidarity  and  prosperity  in  the  EU  by  protecting  and  promoting 
human health and safety and improving public health. Health is a prerequisite for economic recovery 
and ‘inclusive growth’, and the health sector attracts interest for innovation and ‘smart’ investment.  

The Programme financed pan-European actions geared to achieving three main objectives:  

i. 

ii. 

improving citizens’ health security and protecting them from health threats and emergencies, such 
as pandemics and natural disasters; 

promoting health and reducing health inequalities across Europe, whether relating to lifestyle, 
such as access to opportunities for physical activity, to health care, such as access to the necessary 
medical intervention; and 

iii. 

 generating  health  information  and  health  knowledge  and  disseminating  it  to  relevant  parties, 
from the general public to policymakers and health professionals. 

1   Decision No 1350/2007/EC of the European Parliament and of the Council of 23 October 2007 establishing a second programme 

of Community action in the field of health (2008-13) (OJ L 301, 20.11.2007, p. 3–13). 
2   COM(2007) 630 final, 23.10.2007; http://ec.europa.eu/health-eu/doc/whitepaper_en.pdf. 
3   COM(2010) 2020, 3.3.2010; http://ec.europa.eu/europe2020/index_en.htm. 

2 

 
 
  
                                                      
Three  broad  thematic  areas  corresponding  to  these  objectives  were  identified,  with  priorities  and 
sub-priorities (see Figure 1). 

2.2.   What outputs were expected from the Programme?  

Under Article 168 TFEU, the Union shall encourage cooperation between the Member States, and support 
their action, including through the establishment of guidelines and indicators, the organisation of exchange 
of  best  practice  and  support  for  monitoring  and  evaluation.  Member  States’  responsibilities  for  the 
definition of their health policy and for the organisation and delivery of health services and medical care 
should be respected. 

The Programme was expected to support and add value to Member States’ policies and hence contribute to 
protecting the health and safety of citizens through actions in the field of public health. Accordingly, the 
Programme financed a large number of actions with a good coverage of all the priorities and sub-priorities 
established in Decision No 1350/2007/EC, the outputs of which can be broken down as follows:  

-  knowledge-  and  evidence-building  through  studies  and/or  surveys  (e.g.  Eurobarometer),  including 
evaluations and impact assessments that are beneficial on a number of levels, e.g. providing a basis 
for informed policymaking and reporting; 

- 

tools  and/or  methodologies  that  help  to  secure  advantages  for  both  the  public-health  communities 
(e.g. integrating their work processes) and citizens directly (e.g. with regard to improving diagnostic 
tests, improving patient care, etc.); 

-  communication,  awareness-raising  and  networking  (e.g.  co-funding  pan-EU  conferences  and 

networks inter alia in the field of rare diseases); 

-  comparable  data  across  the  EU,  providing  information  for  policymaking  purposes,  e.g. European 

core health indicators (ECHIs);  

- 

training, educational material and guidance with a positive impact on the public-health community 
(e.g. by providing guidelines on patient care, diagnostics, social inclusion of vulnerable groups, etc.) 
and on citizens who might benefit from treatment by better-educated healthcare professionals; 

-  best  practices,  helping  to  achieve  and  maintain  high  standards  in  all  health-related  areas  (research, 

prevention, access, care, treatment, etc.); and 

-  capacity-building in the public-health community at different levels (e.g. increasing the capacity of 
healthcare  systems  to  deal  with  diseases  through  an  exchange  of  knowledge  with  healthcare 
institutions in other Member States). 

3 

 
 
 
Figure 1:   2nd Health Programme – thematic areas, priorities and sub-priorities 

4 

 
 
 
 
2.3.   What results and impacts were expected from the Programme?  

Many health-related challenges, such as cross-border health threats, cannot be addressed at country level; hence 
there  is  a  clear  need  for  EU  action  to  complement  Member  States’  efforts.  However,  any  action  at  EU  level 
should demonstrate EU added value4 and actions co-funded by the Programme were expected to result in one or 
more of the following: 

-  a contribution to the development and/or implementation of EU legislation; 

-  money  saved  and  duplication  of  efforts  avoided  by  cooperation  across  national  health  systems  for  the 

improvement of health in the EU; 

- 

identification  and  application  of  best  practice  in  all  participating  countries,  e.g. procedures,  approaches, 
methods or tools that could be applied by healthcare professionals or others; 

-  evidence-based  decision-making  facilitated,  e.g.  by  providing  scientific  information,  real-time  data  for 

comparison and/or indicators that can inform decision-making at a higher political/policy level; 

- 

- 

risks  reduced  and  consequences  of  cross-border  health  threats  mitigated  by  the  establishing  of  relevant 
structures for coordination; 

increase  in  the  movement  of  patients  and  healthcare  personnel  between  Member  States,  thereby 
contributing to a better match between supply and demand; 

-  sustained  networking  activities  among  stakeholders,  contributing  to  knowledge-sharing  and  health 

capacity-building in the EU; and 

-  support for the deployment of innovative solutions for healthcare provision, in terms of both products and 

services.  

The results overall are expected to impact on public health in Europe in order to achieve the main objective of 
the  Programme,  i.e.  to  complement,  support  and  add  value  to  Member  States’  policies  and  to  contribute  to 
increased  solidarity  and  prosperity  in  the  EU  by  protecting  and  promoting  human  health  and  safety  and 
improving public health. 

2.4.   Baseline 

The  1st  Public  Health  Programme  (2003-2007)  grew  out  of  a  small  number  of  isolated,  empirically  managed 
activities  in  response  to  calls  from  the  Council  and  the  European  Parliament,  such  as  action  on  HIV/AIDS, 
health information, etc. The number of priorities increased gradually around the three main objectives of health 
promotion,  health  security  and  health  information,  in  order  to  optimise  impact  and  meet  new  expectations 
through  an  integrated  approach.  The  Programme  was  operated  exclusively  through  grants  for  projects  and  a 
small number of tenders. The Member States that joined the Union in May 2004 became involved progressively 
and were underrepresented in the actions financed. The Programme was  managed by the Commission, except 
for  a  small  part  which  was  transferred  to  the  Public  Health  Executive  Agency,  which  became  the  Executive 
Agency for Health and Consumers (EAHC) and later the Consumers, Health, Agriculture and Food Executive 
Agency (CHAFEA), after its establishment and operational launch in 2006.  

The  evaluation  of  the  1st  Public  Health  Programme  recognised  its  strong  potential  contribution  to  preparing, 
developing  and  implementing  EU  public-health  policies,  despite  the  broad  spectrum  of  health  priorities  it 
covered, and called for more focus and rationalisation. The dissemination of the results was seen as an important 
area  for  improvement:  the  outcomes  of  the  actions  targeting  health  policymaking  at  EU,  national  or  regional 
levels were neither sufficiently known nor widely used by stakeholders and policymakers. Disseminating results 
was seen as essential to ensuring their sustainability and helping to monitor the impact of the actions.  

The  design  of  the  2nd  Health  Programme  was  similar  to  that  of  its  predecessor,  but  involved  new  financial 
mechanisms, in addition to grants for projects and conferences, in order to respond better to stakeholders’ needs: 
operating  grants  for  non-governmental  organisations,  direct  grants  for  boosting  cooperation  with  international 

4   Following  the  Commission’s  Communication  Reforming  budget,  changing  Europe  in  the  context  of  the  2008/2009  budget  review 
(COM(2007) 1118), ‘EU added value’ was introduced as an award criterion in the evaluation of proposals. On the basis of its experience 
and  expertise,  the  Executive  Agency  for  Health  and  Consumers,  which  the  Commission  entrusted  with  implementation  of  the 
Programme, identified ways in which EU added value is created and methods for assessing it. See also section 7.1 (Relevance and EU 
added value).  

5 

 
 
 
                                                      
health organisations, joint actions with Member States and tenders to cover specific needs related to the support 
of EU health policies.  

3.    EVALUATION QUESTIONS  

The evaluation is based on a set of 14 questions divided into four main areas:  

(a) management tools; 

(b) dissemination practices; 

(c) Programme impact; and  

(d) synergies with other services and programmes.  

The  questions  do  not  follow  the  classical  approach  to  programme  evaluation  (focusing  on  relevance, 
effectiveness, efficiency, coherence and utility) but, following the results and recommendations of the mid-term 
evaluation, target specific areas of concern in programme implementation.  

As regards programme design and management, the mid-term evaluation resulted in recommendations to:  

  define more tangible and focused objectives and establish progress indicators;  

  prepare  strategic  multi-annual  planning  to  determine  appropriate  priority  actions  and  select  the 

corresponding financial mechanisms; 

  provide technical assistance to potential applicants for preparing appropriate proposals; 

  create  a  nomenclature  for  explaining  EU  added  value  and  integrate  it  in  the  application  process  through 

specific criteria; 

  provide further explanations on the scientific evidence required in proposals and how to share it; 

  share other information with Programme stakeholders and potential beneficiaries; 

  develop a regular reporting system for the actions and their results; 

  communicate/disseminate project results better and more systematically and improve communication with 

Programme stakeholders; and  

  make  full  use  of  consistencies  and  complementarities  between  Programme  actions  and  other  actions  at 
international, European and national level, including sharing of data among Commission services, Member 
State authorities and international organisations.  

These  recommendations  underlay  the  first  four  evaluation  questions,  which  were  designed  to  measure  the 
progress made  on the effectiveness of the Programme management: 

EQ 1:   To  what  extent  have  the  recommendations  of  the  mid-term  evaluation  concerning  the  management 

and the design of the Programme been implemented? 

EQ 2:   How effective have recent changes in the emphasis on and use of specific funding mechanisms (i.e. 
use  of  joint  actions,  balance  between  calls  for  proposals  and  calls  for  tender)  been  in  delivering 
policy-related outputs and what was the impact on the geographical distribution of beneficiaries? 

EQ 3:   To what extent did the implementation of previous recommendations influence the Programme’s other 
operations,  including  the  recruitment  of  beneficiaries  and  the  level  of  participation  of  all  Member 
States in Programme actions (including the facilitation of participation from low-GNI countries)? 

EQ 4:   What are the state-of-the-art tools in terms of monitoring project outputs that could be applied to the 
Programme, what are the expected benefits against costs and how could they be implemented? 

As  regards  dissemination  practices,  the  mid-term  evaluation  recommended  fostering  the  dissemination  of 
results  and  organising  an  exchange  of  information  on  results  between  the  Agency,  Commission  officials, 
policymakers in Member States and other stakeholders.  

6 

 
 
 
 
 
The evaluation questions relating to dissemination were:  

EQ 5:   (a) To what extent have the actions/outcomes/results of the 2nd Health Programme been published? 
To  what  extent  are  they  (made)  accessible  to  the  international  scientific  and  health  community,  to 
health 
EU? 
(b) Are the results published and disseminated in a sustainable way?   
(c) How useful is the EAHC database in this context? How can it be improved?  
(d) Which other tools would be useful in this context? 

policymakers, 

the  wider 

society 

public 

civil 

and 

the 

to 

in 

EQ 6:   What is the relation between the publications/activity reporting and Member State participation in the 
2nd Health Programme, the number of health scientists, public-health specialists and physicians per 
Member State? Are patterns identifiable? Have dissemination activities been undertaken in a way to 
overcome possible geographical imbalances in certain actions? 

EQ 7:  To  what  extent  do  stakeholders  other  than  Member  State  governments  (sub-national  regional 
organisations, civil society, social partners, etc.) promote Programme outcomes and results, and via 
which channels? This should consider both organisations funded by the Programme, and others. 

EQ 8:   How could the current dissemination practices be improved to increase return on investment? 

Since the negotiations with the Council and European Parliament on the 3rd Health Programme confirmed the 
important  role  dissemination  plays  in  maximising  programme  impact,  Article  13(4)  of  Regulation  (EU) 
No 282/2014  explicitly  requires  wide  dissemination  of  the  results.5  Accordingly,  these  questions  and  the 
subsequent answers were intended to contribute to improving dissemination further.  

Given the difficulty of assessing the impact of a small programme against the scale of health needs in Europe, 
the  relevant  evaluation  questions  focused  first  on  the  relevance  of  the  Programme  actions  vis-à-vis  the  Union 
mandate  on  health  and  secondly  on  the  short-  and  medium-term  progress  achieved  in  specific  areas.  With  a 
view  to  the  next  programming  period,  the  questions  also  seek  to  elicit  elements  for  a  better  understanding  of 
how the Programme could impact on health policies in the Member States. 

EQ 9:   How and to what extent has the 2nd Health Programme supported Member States’ health policy and 
actions (in relation to the provisions on support, cooperation and coordination in Article 168 of the 
Treaty)? 

EQ 10:   Which are the main health policy areas in which progress has been achieved due to the support of the 

Health Programme, and what constitutes this progress? 

EQ 11:   What are reasonable assumptions on the way to measure the impact of the programme in terms of (a) 
short-term,  (b)  middle-term,  (c)  long-term  timelines  and  (d)  in  relation  to  average  project 
trajectories? 

EQ 12:   Which  factors/reasons  may  intervene  and  positively  or  negatively  influence  the  impact  of  the 

Programme? 

EQ 13:   What are the main lessons than can be drawn to ensure an overall successful transition from the 2nd 

to the 3rd Health Programme? 

The success of the Programme also depends on synergies with other programmes in the area of health. Thus, 
the last evaluation question refers to coherence and consistency and focuses on the two other major programmes 
(under  the  FP7  research  programme  and  the  Structural  Funds)  with  substantial  EU  funding  and  interest  for 
Member  States.  However,  other  synergies  with  smaller  programmes  are  also  covered,  since  the  question 
concerns the Commission’s general objectives for economic growth and social inclusion. 

EQ 14:   What  synergies  are  there  with  other  policies  and  programmes  of  the  Commission  such  as  the 
European Structural and Cohesion Funds, the programmes managed by DG RTD and other DGs (in 
particular  EMPL,  CONNECT)  and  to  what  extent  did  the  Health  Programme  underpin  the 
Commission’s general objectives – focus on Europe 2020 and their objectives related to social policy 
(e.g. the renewed Social Agenda) and economic growth (research and innovation, competitiveness)? 

5  

‘The Commission shall make the results of actions undertaken pursuant to this Regulation publicly available and shall ensure that they 
are widely disseminated in order to contribute to improving health in the Union’; Article 13(4) of Regulation (EU) No 282/2014 of the 
European Parliament and of the Council of 11 March 2014 on the establishment of a third Programme for the Union’s action in the field 
of health (2014-2020) and repealing Decision No 1350/2007/EC (OJ L 86, 21.3.2014, p. 1–13). 

7 

 
 
 
 
                                                      
4.    METHOD  

4.1.   Process and methods used 

The  external  evaluation  study  started  in  May  2014  and  the  final  report  was  delivered  in  July  2015.  An 
interservice  steering group established in December 2013 discussed and validated  the  evaluation  mandate  and 
agreed  on  the  evaluation  questions  and  the  terms  of  reference6  for  the  specific  contract.  The  group  met  four 
times to discuss in addition to the above, the inception, interim and draft final reports and provided comments on 
the  methods  and  organisation  of  the  evaluation.  It  was  composed  of  representatives  from  DG  SANTE,  RTD, 
AGRI,  JRC,  EMPL,  REGIO,  EAC  and  CONNECT,  the  Secretariat-General  and  CHAFEA.  In  addition,  the 
European Hospital and Healthcare Federation (HOPE), the European Health Management Association (EHMA) 
and the European Public Health Association (EPHA) were represented in order to feed in the views of health-
policy stakeholders. In the spirit of a collaborative and transparent approach, the findings and main conclusions 
were presented to the national focal points on 12 January and 22 May 2015 and to the Programme Committee 
members on 6 March 2015 and 4 February 2016 and they were asked for their comments and opinions. 

The  evaluation  involved  a  variety  of  quantitative  and  qualitative  data  collection  and  review  methods  and 
analytical  tools  to  respond  to  specific  information  needs  and  requirements  respecting  the  principle  of 
triangulation.  Annex  I  contains  a  matrix  showing  the  various  tools  used  to  make  assessments  on  the  basis  of 
agreed judgment criteria and answer each of the evaluation questions. The contractors used desk research, direct 
observations, a survey of the national focal points and interviews with Commission officials, CHAFEA project 
officers, Programme Committee members, beneficiaries and project leaders, and the assistants of two Members 
of  the  European  Parliament  to  generate  data  for  analysis.  Also,  they  carried  out  an  analysis  of  80  actions 
selected proportionally from across the main Programme areas, priorities and financial mechanisms to assess EU 
added value and to review the type of actions (research, development and implementation), the type of partner 
organisations, the partnerships’ geographical spread, cross-sectoral cooperation and dissemination practices.  

To  assess  how  co-funded  actions  contributed  to  the  Programme  objectives  and  identify  factors  that  could 
strategically  maximise  the  potential  impact  of  the  Programme,  the  evaluators  selected  13  case  studies  (five 
projects, five joint actions and three tenders) from the 80 actions in order to delve deeper into specific aspects, 
such as the design of actions, implementation, results and their dissemination, and added value.  

They  also  conducted  a  bibliometric analysis  of  the Programme’s  visibility in  scientific  journals  and,  to  some 
extent, an assessment of Member States’ public-health capacity in relation to their capacity to participate in 
the Programme and make use of the funding.  

Finally,  on  the  basis  that  it  is  critical  for  the  Programme’s  success  to  ensure  that  all  key  stakeholders  are 
effectively engaged in and/or informed of the Programme and its results, the contractors undertook an analysis 
of  the  Programme  stakeholders.  This  sought  to  explore  the  power,  position  and  interests  that  different 
stakeholder groups brought to the Programme and to identify how they could be involved further. 

When  carrying  out  the  work,  the  contractors  defined  conditions  and  features  on  the  basis  of  which  to  assess 
Programme actions and verified their validity, in particular through the case studies: 

  essential  conditions,  common  to  all  actions,  that  influence  effectiveness  and  thus  could  influence  the 

probability of the action having an impact in the longer term (Table 1); and 

  specific key features per funding instrument, as each instrument is meant to respond to different needs and 
produce  certain  outputs/results,  e.g. tenders  to  obtain  studies  and  respond  to  specific  Commission  needs, 
joint actions to boost Member States’ cooperation on common health issues, calls for proposals for projects 
on health issues with a wider scope and to incentivise innovation, and operating grants to support NGOs 
and specific networks. 

6   The terms of reference are set out in section 1 of the annexes to the evaluation report. 

8 

 
 
 
 
 
                                                      
Table 1:   Essential conditions for actions’ effectiveness 

Condition 

Description 

The actions address a relevant problem 

To contribute to HP objectives, actions need to address a specific problem that fits into one 
of the priorities and where the EU added value of action is high. 

The  actions  are  based  on  concrete  and 
SMART objectives. 

In  order  to  implement  a  service  efficiently,  you  need  to  develop  concrete  objectives, 
operationalised  in  a  SMART  way,  i.e. your  objectives  should  be  specific,  measurable, 
achievable, realistic and time-dependent. 

The actions are evidence-based. 

The actions have clear target groups. 

Overall and SMART objectives are more likely to be achieved when projects are designed 
on  an  evidence-based  understanding  of  how  the  activities  they  implement  are  related  to 
what they actually want to achieve. This includes building on existing knowledge. 

Efficient organisations are often characterised by a relevant and explicit definition of their 
target groups. 

The  actions  have  developed  adequate 
implementation strategies 

The goals of the HP are far-reaching and require dedicated effort over a long period. This 
means that the chances of achieving long-term effects improve if the project activities are 
sustainable and are implemented by the participating actors. 

The  actions  are  characterised  by  a  high 
degree of target achievement. 

If the above conditions are met and actions achieve what they set out to do, this is likely to 
generate impacts that contribute to the wider HP objectives.  

The  actions  have  effective  strategies  for 
disseminating results. 

Dissemination  of  results  is  key  to  facilitating  their  take-up  beyond  the  participants 
themselves. 

The  contractors  sought  to  take  an  innovative  approach  to  assessing  on-going  efforts  (in  line  with  the 
recommendations of the final evaluation of the 1st Public Health Programme and the mid-term evaluation of the 
2nd  Health  Programme)  to  involve  low-GDP/GNI7  Member  States.  This  meant  measuring  not  only  increased 
participation in calls, but also initiatives for the transfer of knowledge to these countries (evaluation questions 3 
and 6). The contractors suggested approaching participation by countries that joined the EU from 2004 onwards 
not  only  from  an  economic  angle,  i.e.  participation  rates  of  low-  versus  high-GPD/GNI  countries,  but  also 
assessing the relationship between countries’ participation and their ‘public-health capacity’.   

4.2.   Limitations – robustness of findings 

The  evaluation  is  not  based  on  a  theory-change  approach8,  as  it  was  considered  too  difficult  to  construct  a 
posteriori  an  overall  intervention  logic  for  a  programme  with  very  broad  objectives  and  multiple  priorities 
grounded in the EU’s supporting competence in public health, as laid down in the Treaty on the Functioning of 
the European Union. It was also too early to assess results as outputs of the actions were just being delivered. 

For  this reason,  the  contractors  based  their  work on  explicit  expectations  and assumptions  (see  the  essential 
conditions  and  specific  key  features  above)  as  to  what  the  Programme  and  the  various  financial  mechanisms 
were  to  achieve.  They  followed  a  purpose-driven  approach to  sampling  (for  the  in-depth  review,  case  studies, 
bibliometric  analysis,  stakeholder  analysis,  etc.),  focusing  on  those  actions  and  facets  of  the  Programme  that 
promised to be of most value and interest for the analysis, given the specific evaluation purpose and questions. 
The Commission services gave their agreement to the choices made.  

It  was  also  decided  to  limit  the  breadth  of  the  evaluation,  since  the  3rd  Health  Programme  had  already  been 
launched and certain aspects became more pertinent than others, depending on their continued relevance for the 
new Programme (see point 2.2 on the scope of this evaluation). As a result, while certain key features of the 2nd 
Programme  (in  particular  the  very  broad  scope  and  resulting  lack  of  focus,  including  operational  and  specific 
objectives) would normally have been addressed in more depth in a final evaluation, they were not given much 
prominence  here.  Relevant  recommendations  (e.g. as  regards  the  need  for  more  specific  objectives  and 
indicators)  had  already  been  made  in  the  mid-term  evaluation  and  addressed  in  the  design  of  the  3rd  Health 
Programme. 

As  mentioned  above,  the  contractors  went  beyond  distinguishing  participation  between  high-  and  low-GNI 
Member States to explore the statistical relationship between Member States’ ‘public-health capacity’ and their 

7   Gross domestic product (GDP); gross national income (GNI). 
8    A theory of change is a tool for developing solutions to complex social problems. A basic theory of change explains how a group     of 

early and intermediate accomplishments sets the stage for producing long-range results. 

9 

 
 
 
 
                                                      
 
participation  rates  (taking  as  a  proxy  variable  the  amount  of  funding  that  organisations  from  a  given  country 
were able to obtain). This approach proved to have significant methodological limitations, mainly due to the lack 
of  a  commonly  agreed  definition  of  ‘public-health  capacity’.  Consequently,  the  analysis  was  limited  to  some 
indicators  relating  to  wealth  (GDP/GNI),  health  research  spending,  health  expenditure,  health  publications, 
healthcare  resources,  health  outcomes  and  healthcare  performance.  Moreover,  data  quality  and  availability  for 
some of these were not always ideal for the correlations that the evaluators were examining. 

The  weaknesses  in  the  design  of  the  Programme  objectives  resulting  in  the  lack  of  indicators  for  systematic 
monitoring as already found in the mid-term evaluation posed another limitation to assess the effectiveness of 
the programme. 

Given  these  limitations,  the  findings  and  conclusions  are  representative  only  for  the  actions  investigated  and 
cannot necessarily be extrapolated to all actions under the Programme, as the results of individual actions cannot 
simply be aggregated to assess overall impact.  

Also,  given  the  broad  Programme  objectives,  limited  dissemination  efforts,  dependence  on  the  willingness  of 
Member State authorities to take up the results and integrate them in national health policies, and the time taken 
for health indicators to change, the evaluation can at this stage assess outputs from individual actions only and 
not the impact of the whole Programme. 

4.3.   Quality assessment of the study 

By and large, we agree with the contractors’ findings, the answers to the evaluation questions and its conclusions 
within the limitations described above9. In the contractors’ discussions with interservice steering group members, 
it became clear that, for four questions10 more information would have been appreciated.  

In relation to the effectiveness of changes concerning the specific funding mechanisms (i.e. use of joint actions, 
balance  between  calls  for  proposals  and  calls  for  tender)  in  delivering  policy-related  outputs  the  contractors’ 
approach  is  more  theoretical  and  explains  how  it  was  expected  that  the  Programme  would  support  Member 
States’ health policies, but it was not possible to show, on the basis of outputs to date, the extent to which it has 
achieved its goal. 

In relation to the identification of patterns between Member States participation in the 2nd Health Programme 
and  its  public  health  capacity  the  limited  quality  and  availability  of  data  on  Member  States’  public-health 
capacity did not allow for a robust analysis.  

Suggestions for improvement on dissemination practices were limited to better targeting audiences but did not 
look into return on investment as required in the Terms of Reference. We accepted the contractor's explanation 
that it will not be feasible to measure the ‘return on dissemination investment’ in terms of health outcomes, since 
this would require complex models to assess evidence-based policy-making, which are outside of the scope of 
the assignment. 

In the absence of explicit specific objectives and indicators already from the design of the 2nd Health Programme, 
it was not possible for the contractor to clearly indicate the extent to which the Programme supported Member 
States’  health  policies.  The  Programme  is  a  series  of    successful  individual  actions  but  it  is  impossible  for 
numerous reasons to draw concrete links from individual actions or the Programme as a whole to the high-level 
indicators ( i.e. Healthy Live Years) relating to health outcomes, when these are sometimes available. Also the 
actions’  desired  outcomes,  even  in  the  best  circumstances,  take  years  to  materialise  and  are  largely  highly 
specific to the actions in question. 

9   The quality assessment of the ex post evaluation is provided in Annex IV of this document.  
10   EQ 2, where the contractors’ approach is more theoretical and explains how it was expected that the Programme would support Member 
States’ health policies, but that it is not possible to show, on the basis of outputs to date, the extent to which it has achieved its goal; 

EQ 6, where the limited quality and availability of data on Member States’ public-health capacity undermined the patterns identified; 

EQ  8,  where  suggestions  were  made  to  tailor  dissemination  to  specific  target  audiences  taking  into  consideration  the  stakeholder 
analysis, but without looking into return on investment, a question raised specifically in relation to the likely costs of dissemination; and

EQ 9, where the contractors do not indicate clearly the extent to which the Programme supported Member States’ policies and actions. 

10 

 
 
 
 
                                                      
 
 
 
 
 
 
5.    IMPLEMENTATION OF THE PROGRAMME 

The Commission prepares every year, in close consultation with Member States health authorities serving on the 
Programme Committee the Annual Work Programme and adopts it through "Comitology" procedure. The Work 
Programme defines the most relevant actions to address Member States health needs and create added value at 
EU level. These actions should have high public health relevance and pertinent geographical coverage.  

The CHAFEA11 was entrusted with implementing most of the Programme through competitive calls for grants 
and  tenders.  The  Commission  implemented  only  specific,  highly  policy-relevant  service  contracts  and 
cross-cutting actions, such as IT services, itself.  

6.1.  

Budget distribution per financial mechanism 

Various financial mechanisms were used to implement the Programme: 

  Projects  are  used  to  explore  a  wide  range  of  subject  areas  and  delivery  mechanisms,  and  take  forward 
health  policy  initiatives  in  an  innovative  way,  almost  as  ‘pilots’.  They  absorbed  most  of  the  available 
budget  and  provided  significant  scope  for  innovation.  Their  use  declined  in  the  second  half  of  the 
Programme, mainly in favour of joint actions and tenders, in an effort to concentrate the Programme on a 
series of a few major actions aligned with the Europe 2020 strategy;  

  Tenders: calls for service provision are used to cover specific Commission needs with regard to studies, 
evaluations,  surveys  and  technical  assistance.  This  includes  IT-  and  communication-related  services 
required to develop and update EU health legislation or to fulfil the Commission’s obligations under EU 
health legislation. A good example is the development of reference tools for the design and use of a single 
European  coding  system  for  tissues  and  cells.12  Service  contracts  are  also  used  where  the  scope  and 
objectives  are  very  concrete  and  under  Commission  control,  e.g. in  the  development  and  conduct  of 
training courses and exercises with Member States to build capacity to deal with emergency situations; 

  Joint  actions are  a  new  financial  mechanism  introduced  in  the  2nd  Health  Programme  to  cover  specific 
health-policy  needs  and  aimed  at  supporting  EU  cooperation  with  as  many  partners  as  possible  from  all 
countries  participating  in  the  Programme,  to  generate  momentum  for  wider  impact.  The  number  of  joint 
actions called for increased from 2011 onwards in order to enhance the Programme’s policy relevance and 
make it more compatible with the Europe 2020 objectives of smart and inclusive growth. 

Joint  actions  are  often  started,  after  several  years  of  cooperation  between  relevant  stakeholders  from  (or 
designated  by)  Member  State  authorities,  in  a  bid  to  secure  political  endorsement  and  optimise  policy 
coordination. They typically develop/share/refine/test tools, methods and approaches for specific issues or 
activities and involve a degree of capacity-building. . The gain for the Member States involved is expected 
to  be  substantial  in  terms  of  knowledge  and  experience  exchanged  and  should  also  lead  to  tangible  cost 
savings.  For  this  reason,  the  Programme  sought  to  ensure  that  joint  actions  attract  the  widest  possible 
participation from all Member States;  

  Operating grants are also a new instrument in the 2nd Health Programme. They support the running costs 
of  pan-European  NGOs  and  specific  networks  that  focus  on  priority  health  issues  and  contribute  to 
furthering health policy in the EU;  

  Conference grants: while support for pan-EU conferences on important health topics was not really new 
as a type of action, their selection through an annual competitive call, separate from the call for projects, 
was  introduced  under  the  2nd  Health  Programme  to  avoid  competition  with  proposals  for  larger  health 
projects.  Grants  for  (twice-yearly)  central  conferences  on  health  were  awarded  directly  to  the  Member 
State holding the Presidency, which also selected the conference topic and took care of the organisation; 
and 

  Direct  grants  to  international  organisations,  such  as  the  World  Health  Organisation  (WHO)  and  the 
Organisation  for  Economic  Cooperation  and  Development  (OECD),  were  provided  to  continue 
international cooperation on major health issues (mainly the collection and analysis of health data).  

11   Previously EAHC (Executive Agency for Health and Consumers) 

12   http://ec.europa.eu/chafea/health/tenders_H03_2011.html; 

http://ec.europa.eu/health/blood_tissues_organs/docs/tissues_single_european_code_en.pdf. 

11 

 
 
 
                                                      
In total, 788 actions were financed: 147 projects, 30 joint actions, 420 service contracts, 84 operating grants, 36 
direct  grants  with  international  health  organisations  and  71  conferences.  The  overall  budget  distribution  per 
funding mechanism is shown in Table 2. 

Table 2:   Programme spending by funding mechanism 

Funding mechanism 

Total  

Projects  

Service contracts (tenders) 

Joint actions 

Operating grants 

Direct grant agreements 

Grants for conferences 

Other13 

Total 

 €106 293 671.24  

 €72 053 873.45  

 €63 962 704.38  

 €20 825 185.85  

 €13 805 987.00  

 €5 268 308.14  

€11 693 227.81 

% 

36 % 

25 % 

22 % 

7 % 

5 % 

2 % 

4 % 

€293 902 957.87 

100 %14 

Source: CHAFEA database and DG SANTE 

The financial instruments used most were projects, service contracts and joint actions; together, these accounted 
for more than 80 % of the budget.  

6.2.  

Budget distribution per thematic area 

In  pursuit  of  its  objectives,  the  Programme  supported  actions  in  three  thematic  areas:  health  security,  health 
promotion and health information. Actions supporting the objective of health promotion were at the heart of the 
Programme,  accounting  for  57 %  of  total  funds  allocated,  while  the  areas  of  health  security  and  health 
information received 23 % and 21 % respectively.15 

13   ‘Other’  includes  actions  signed  and  committed  to  by  DG  SANTE  and  CHAFEA,  such  as  special  indemnities  to  experts  for  their 
participation in and work for EU scientific committees, an administrative agreement with the Joint Research Centre, publications and 
various communication initiatives to promote the 2nd Health Programme, sub delegations to Eurostat, etc. 

14   Figures do not add up to 100 % due to rounding. 
15   Due to rounding, these percentages do not add up to 100 %. Operating grants are included in this attribution of funds per strand. 

12 

 
 
 
 
 
                                                      
Figure 2:  1st Public Health Programme and 2nd Health Programme spending by thematic area 

Source: CHAFEA database and DG SANTE 

Health  promotion  gained  in  prominence  as  compared  with  the  1st  Public  Health  Programme  (see  Figure 2), 
underlining  the  importance  the  2nd  Health  Programme  placed  on  addressing  health  determinants  and  tackling 
health  inequalities.  Meanwhile,  actions  focusing  on  the  generation  and  dissemination  of  health  information 
declined.  The  relative  importance  ascribed  to  the  health  security  objective  remained  virtually  unchanged, 
although  the  epidemiological  surveillance  networks  were  transferred  at  the  end  of  the  1st  Public  Health 
Programme to the European Centre for Disease Prevention and Control (ECDC). 

Overall,  the  contractors  found  that  five  priorities  and  sub-priorities  accounted  collectively  for  about  three 
quarters of Programme spending: 

  health determinants and healthy lifestyles: 24 % of the overall Programme budget was aimed at tackling 
key health determinants such as nutrition, alcohol, tobacco and drugs, and other determinants more related 
to social and environmental factors; 

  prevention  of  major  and  rare  diseases:  16 %  of  the  overall  spending  related  to  major  diseases 
(e.g. cardiovascular  disease,  cancer  and  HIV/AIDS  prevention)  and  rare  diseases  (including  support  for 
developing  recognised  expert  reference  groups,  assistance  to  Member  States  in  developing  and  taking 
forward  rare-disease  strategies,  and  contributing  to  WHO  international  classifications  of  rare  diseases); 

  health  monitoring  and  data:  11 %  of  the  budget  was  spent  on  forming  an  effective  and  sustainable 
network  for  health  technology  assessment  (HTA)  across  Europe  to  help  develop  reliable,  timely, 
transparent  and  transferable  information  to  contribute  to  HTAs  in  European  countries.  It  also  supported 
work relating to ECHIs to facilitate monitoring and comparison between EU countries, thereby serving as a 
basis for policymaking; 

  health  threats:  13 %  for  actions  inter  alia  to  facilitate  collaboration  between  laboratories  and  develop 
common  testing  methods,  with  the  aim  of  developing  strategies  and  mechanisms  to  respond  to  health 
threats and emergencies; 

  safety: 10 % to fund a variety of actions relating to issues such as organ donation and transplantation, and 
patient safety, some of which facilitated the exchange of organs donated in Member States; assessing data 
on manufactured nanomaterials and seeking to establish a European framework for the evaluation of organ 
transplant results. 

13 

 
 
 
 
  
 
  
  
  
The  reports  and  deliverables  produced  by  the  actions  co-funded  under  the  Programme  are  available  on 
CHAFEA’s website. A database16 provides open access to the results, with the exception of service contracts and 
direct agreements with international organisations.  

6.3.  

Participation by type of stakeholder group 

A  wide  and  diverse  range  of  stakeholders  participated  in  and  benefited  from  the  grants  provided  by  the 
Programme. Figure 3 provides a breakdown by group of grant beneficiaries.17 More specifically, the groups are 
as follows: 

  government organisations: these represent the largest group (37.8 %) of stakeholders that participated in 
the  Programme.  They  include  health  policymakers,  regulators,  general  or  specialised  governmental 
public-health  organisations  and  institutions,  and  healthcare  providers.  They  were  chiefly  interested  in 
participating in joint actions, particularly those relating to health security and health information; 

  non-profit  and  non-governmental  organisations:  these  make  up  a  second  rank  of  participating 
organisations (30.7 %). They mainly received operating grants, but also participated in projects and joint 
actions; 

  academic and research organisations: with a share of 26.5 %, these were involved mainly in projects (to 

the same approximate extent across all three thematic areas); 

  commercial  organisations:  their  participation  accounts  for  3.2 %  (significantly  more  if  we  take  into 
consideration their participation in tenders which are in the most of the cases addressed to them); and  

 

international organisations: their cooperation with the Commission, mainly under the health information 
objective (collecting and analysing health data), accounted for 1.7 % of the total.  

Figure 3:   Participation of stakeholder groups receiving grants under the 2nd Health Programme 

Source: CHAFEA database and DG SANTE 

6.4.  

Participation by geographical area  

All Member States and the three EEA EFTA countries (Iceland, Liechtenstein and Norway) participated in the 
Programme. As the Programme was open to candidate countries, Croatia was involved for the entire period (as a 
Member State from 1 July 2014). In line with recommendations in the final evaluation of the 1st Public Health 
Programme,  efforts  were  made,  especially  through  the  joint  actions,  to  involve  more  actors  from  the  ‘EU-12’ 
Member States that joined the Union in and after 2004.  

16   http://ec.europa.eu/chafea/projects/database.html. 
17   This figure covers all funding except tenders. 

14 

 
 
 
 
                                                      
In  terms  of  number  of  beneficiaries,  participation  in  the  Programme  reflected  relative  population  sizes.18  Just 
over three quarters (76.1 %) of beneficiaries were based in the ‘EU-15’, while the EU-12 accounted for 20.4 %. 
There is a disparity in terms of the allocation of funding, with 88.3 % going to organisations from the EU-15 and 
9.4 %  to  those  based  in  the  EU-12.  This  is  probably  due  in  part  to  differences  in  wages  and  labour  costs. 
However, the difference between EU-15 and EU-12 is far more pronounced when one considers the spread of 
lead beneficiaries, of which an overwhelming 95 % were based in the EU-15, with only 4 % based in the EU-12. 
This was especially visible in service contracts, projects and operating grants, for which nearly all lead partners 
were based in the EU-15. However, 15 % of lead partners for grants for conferences (11 of 71) and 10 % for 
joint actions (3 of 30) were based in the EU-12. This breakdown is presented in Figure 4.  

Figure 4:   Proportion  of  (total  and  lead)  beneficiaries  and  funding  received  (EU-15,  EU-12  and  other 

participating countries) 

Source: CHAFEA database and DG SANTE 

The mid-term evaluation mentioned that administrative and cultural barriers, in addition to financial constraints, 
could  act  as  obstacles  to  the  participation  of  low-GDP  countries.  The  final  evaluation  (following  a  survey  of 
national  focal  points)  reported  that  the  EU-15  faced  more  (or  at  least  cited  more  often)  a  lack  of  human  and 
financial resources and administrative burden (see Figure 5).  

Figure 5:   Main barriers to the participation of EU-12 and EU-15 organisations 

Source: Survey of national focal points in the framework of the ex post evaluation 

18   According  to  Eurostat  data,  in  2011  the  total  EU  population  was  501  million,  of  whom  404  million  (81 %)  lived  in  the  EU-15  and 

97 million (19 %) in the EU-12. 

15 

 
 
 
 
 
 
 
                                                      
6.   FINDINGS OF THE EVALUATION 

7.1.  

Programme management 

The relevance of the Programme was extensively assessed by the mid-term evaluation, but the ex post evaluation 
provided a second opportunity to examine the relevance of a series of co-funded actions. This was done on the 
basis of the 13 case studies, which led to positive conclusions.  

The mid-term evaluation concluded that the Health Programme is focusing on relevant priority areas addressing 
the main public health issues in Europe; however the Programme's broad objectives were not helpful to prioritise 
actions as most health-related issues could fit under them under any circumstances. 

However,  the  ex-post  evaluation  also  found  that  during  its  second  half,  the  Health  Programme  increased  the 
policy  relevance  of  funded  actions.  Through  an  increased  involvement  of  DG  SANTE's  management  in  the 
annual planning a greater level of coherence with the Europe 2020 targets was achieved. Through making use of 
joint  actions  to  a  greater  extent  buy-in  from  national  governments  and  participation  of  key  stakeholders  from 
nearly all Member States was secured. However, the lack of appropriate indicators at Programme level and the 
absence of a systematic monitoring to link the available data at action level with higher level health indicators 
made it difficult to fully understand whether and how the Programme impacts on national health policies. 

Selecting actions on the basis of their EU added value 

The 2nd Health Programme aims at complementing, supporting and adding value to Member States’ policies thus 
providing  ‘EU  added  value’.  As  Decision  No 1350/2007/EC  establishing  the  2nd  Health  Programme  did  not 
define ‘EU added value’, the EAHC/CHAFEA developed seven criteria to determine whether proposed actions 
have the potential to generate EU added value. These criteria were tested and validated in the course of the mid-
term evaluation in 2011. The Commission added an eighth criterion concerning potential for innovation in the 
area  of  health  and  integrated  all  eight  criteria  in  Regulation  (EU)  No 282/2014  (recital  6)  establishing  the  3rd 
Health  Programme  and  in  the  subsequent  awarding  procedures  for  actions  to  be  co-funded.  The  eight  criteria 
used are the following:  

1)  implementing EU legislation: to ensure that actions contribute to the development and/or implementation 

of EU legislation; 

2)  economies  of  scale:  to  save  money  and  provide  a  better  service  to  citizens  by  avoiding  duplication  of 

effort and cooperating across national health systems; 

3)  promotion of best practice: to apply best practice in all participating Member States, e.g. by identifying 
procedures, approaches, methods or tools that could be applied by healthcare professionals or others; 

4)  benchmarking  for  decision-making:  to  facilitate  evidence-based  decision-making,  e.g. by  providing 
scientific information, real-time data for comparison and/or indicators that can impact decision-making at 
a higher political/policy level; 

5)  cross-border  threats:  to  reduce  risks  and  mitigate  the  consequences  of  cross-border  health  threats  by 

establishing relevant coordination structures; 

6)  free  movement  of  persons:  to  increase  the  movement  of  patients  and  healthcare  personnel  between 

Member States, thereby contributing to a better match between supply and demand; 

7)  networking: 

to  ensure 

that  networking  activities  among  stakeholders,  which  contribute 

to 

knowledge-sharing and building health capacity in the EU, are supported and sustained; and 

8)  unlocking  the  potential  of  innovation:  to  support  the  deployment  of  innovative  solutions  for  healthcare 

provision, in terms of both products and services. 

In the ex-post evaluation 80 actions were scored by an expert panel for potential to deliver EU added value and 
under which added value criteria (validation of the selection process). For 13 actions outputs and results were 
assessed  for  delivering  EU  added  value.  This  served  as  a  basis  for  analysing  which  type  of  actions  have  the 
greatest potential to deliver EU added value and on which criteria added value is being delivered. 

The  evaluation  found  that  use  of  the  criteria  is  effective  and  that  actions  co-funded  through  the  Programme, 
particularly  the  joint  actions,  scored  high  on  the  EU  added  value.  Much  of  the  demonstrable  EU  added  value 

16 

 
 
 
relates  to  the  identification  of  best  practices,  the  scientific  evidence  to  be  used  for  benchmarking  for 
decision-making and networking activities (see figure 6 and 7). However, these three criteria are not sufficiently 
linked  to  tangible  and  concrete  benefits.  Further  guidance  on  these  criteria  would  be  necessary  to  enable 
applicants to propose more suitable actions that not only identify good practices, for instance, but also address 
barriers  to  implementing  them  across  the  EU.  Actions  received  medium  scoring  for  innovation,  EU  health 
legislation and economics of scale while the criteria of cross-border health threats and free movement of persons 
were under -represented. 

Figure 6: Average scores by EU added value criteria, all actions19 

Figure 7: Proportion of actions averaging scores of 2.0 or more and 1.0 or less, by EU added value criteria 

This way, the evaluation made clear that the robustness and completeness of the three first highly scored criteria 
is  not  optimal  and  lack  the  necessary  discrimination  power  to  avoid  that  the  large  majority  of  the  actions  fit 
broadly under these ones. 

19 

 The  scoring  scale  ranged  from  0-3  as  follows:  0  indicated  ‘no  EU  added  value  foreseen’;1  indicated  ‘EU  added  value  possible’;  2 
indicated ‘EU added value likely’ and 3 indicated EU added value almost certain. 

17 

 
 
 
 
 
                                                      
Effectiveness and efficiency of the Programme management 

The  effectiveness  of  the Programme  management  was  assessed by  the contractors,  including  the programme's 
increased focus, on priority areas, while addressing Member States’ needs and encouraging their participation. 
Consideration  was  given  to  how  implementation  was  monitored  and  how  results  were  disseminated.  Relevant 
findings will be used to inform implementation of the current 3rd Health Programme on the basis of the findings 
of the contractor, the following conclusions can be drawn. 

The Programme  management improved significantly in the second half of the period. Substantial efforts were 
made  to  implement  recommendations  from  the  mid-term  evaluation,  relating  inter  alia  to  more  strategic 
programming20, the systematic use of EU-added-value criteria in grant applications and evaluation21, providing 
clearer guidance to  applicants  and  having better  contact with  applicants and beneficiaries22.  Annex II  presents 
the measures taken to implement these recommendations and the remaining issues looked into in the evaluation 
with a view to suggesting further improvements.  

The changes in the management of the Programme increased its potential to serve Member States’ needs and to 
complement  and  support  their  health  policies.  More  directive  planning  methods  and  increased  use  of  joint 
actions and tenders resulted in a greater focus on specific health policies in order to meet specific needs.  

While the 1st Public Health Programme was implemented mainly via projects and a small number of tenders, the 
2nd  Health  Programme  relied  increasingly  on  joint  actions.  These  aim  to  involve  a  significant  number  of 
Member  States  working  jointly  on  key  health  policies  in  the  expectation  that  the  outputs  will  be  more 
policy-related (as compared with outputs from projects). A total of 30 joint actions were co-funded during the 
2nd Health Programme, for almost €64 million. A full list of joint actions and overall achievements is provided 
in Annex III.   

Although  joint  actions  attracted  participation  from  all  Member  States  and  other  participating  countries,  the 
Programme  does  not  seem  geographically  balanced  in  terms  of  budget  distribution  and  Member  States’ 
participation (total number of beneficiaries and beneficiaries in leading positions), as shown in Figure 4. While 
joint actions were a financing mechanism used increasingly in the second half of the Programme and attracted 
relatively more participants from the EU-12 countries (which accounted for 13 % of the overall budget spent on 
joint actions),23 the intensive use of calls for tender over the Programme period attracted interest chiefly from 
entities in a limited number of EU-15 Member States, with Belgium in the lead. This offsets the relatively higher 
EU-12 participation in joint actions. 

This evaluation takes a significant step forward in starting to reflect on Member States’ public health capacity 
and how this affected their participation in the Programme. The previous evaluations have only mentioned the 
financial  barriers  some  Member  States  could  face  for  their  participation  in  the  Health  Programme,  and  the 
distinction was made between high and low GDP/GNI Member States. The ex-post evaluation even under the 
methodological  limitations  imposed  by  the  fact  that  there  is  no  common  agreed  definition  at  EU  level  for 
"public-health  capacity",  gave  a  relatively  interesting  insight  using  only  some  indicators  relating  to  wealth 
(GDP/GNI),  health  research  spending,  health  expenditure,  health  publications,  healthcare  resources,  health 
outcomes and healthcare performance.  It permitted the linkage of the low participation of Member States (in the 
most  of  the  cases  these  are  low  GNI  countries)  with  what  public-health  capacity  meant  for  the  Health 
Programme. Building public health capacity was not an objective under the Programme, but findings suggest a 
posteriori that it may have been a means or even a prerequisite for successful participation: the Member States 
that  participated  more  actively  were  those  with  solid  public-health  capacity,  while  the  ‘weaker’  countries  had 
lower participation rates and received less funding. 

20   Annual Work Programmes of 2011 and onwards have streamlined the number of priorities and proposed actions in coherence with the 

Europe 2020 Strategy (see below,  table 3)  

21   EU added value criteria were put to systematic use during the second half of the HP, in particular being built into the application and 
assessment process for actions. Definitions of EU added value criteria were provided in the FAQ for the final year of calls for proposals 
for the 2nd HP (2013). For the 3rd HP, EU added value criteria are enshrined in the Programme Regulation 282/2014/EC, included in 
the 2014 AWP and references included in guides for applicants. 

22     See annex II, "Recommendations applied" and more specifically in p. 39 where it is mentioned that a guidance document for actions 
developed by CHAFEA and guide available for each funding mechanism together are available each year with the call for proposals;  
Positive feedback from survey of applicants to calls for proposals 2008  – 2013 regarding both the guidance documents and helpdesk 
services but still room for improvement (for example Frequently Asked Questions section should contain more technical answers rather 
than mainly general ones).  

23   For projects, the EU-12 share of the budget was 11 %. 

18 

 
 
 
                                                      
The evaluations repeatedly point to the challenge of monitoring Programme implementation. Monitoring output 
and outcome from a programme with so many actions, diverse in terms of form and content, is not always easy, 
especially when it comes to making links to higher-level public-health indicators. However, CHAFEA collects 
comprehensive monitoring data at input and activity levels. However the data are not organised systematically; 
which hampers their aggregation and use in real time to inform strategic planning so as alignment with priorities 
and  objectives  cannot  sufficiently  be  ensured  and  deficiencies  persist  in  the  monitoring  of  the  Programme 
performance.  Given  the  architecture  of  the  Programme,  the  evaluation  was  unable  to  shed  more  light  on  this 
issue  or  provide  a  common  set  of  indicators  for  all  actions  or  objectives.  It  does,  however,  underline  the 
importance  of  suitable  indicators  for  good  monitoring  and  reporting  for  the  improved  dissemination  of  action 
outputs. 

Dissemination activities 

The increased dissemination efforts are recognised as contributing to the success of the Programme. Failure to 
share  outputs  and  results  with  those  who  need  them  to  build  health  policies  and  other  initiatives  based  on 
scientific knowledge tested in real settings constitutes a real obstacle to assessing the Programme’s impact.  

The contractors carried out an extensive analysis of the means used for dissemination, either by the beneficiaries 
in  the  framework  of  the  co-funded  actions  or  by  CHAFEA,  on  a  more  aggregated  level.  Previous  evaluations 
showed that Programme outputs and results should be disseminated at three levels: 

1)  at  the  level  of  the  co-funded  actions;  every  action  has  its  own  dissemination  strategy  and  plan,  which  in 
some instances is very effective and in others less so, mainly due to a lack of clarity and focus as to the 
most relevant target audiences and how best to reach them;  

2)  dissemination  activities  organised  and  means  produced  by  CHAFEA,  such  as  brochures,24  ‘cluster 

meetings’,25 project database,26 etc.; and  

3)  dissemination  by  the  Commission,  e.g.  a  high-level  conference  on  EU  health  programmes  organised  in 
Brussels  on  3  May  2012,27  DG  SANTE’s  bi-monthly  electronic  newsletter28  and  information  for 
policymakers, Programme Committee members, the European Parliament and the Council through annual 
programme implementation reports.29 

The  contractors  assessed  these  levels  and  the  means  applied  in  more  detail.  For  example,  beneficiaries  are 
encouraged  to  publish  their  results  in  scientific  journals.  The  bibliometric  analysis  showed  that  numerous 
published  articles  (more  than  expected)  referred  to  2nd  Health  Programme  actions,  but  the  visibility  of  the 
Programme  is  not  always  sufficient,  as  it  was  not  always  acknowledged  as  the  source  of  the  funding,  even 
though this is required under the grant agreements. 

As  the  most  appropriate  audiences  for  the  dissemination  of  results  vary,  so  do  the  most  effective  tools  and 
channels  for  reaching  these  audiences.  Some  actions  and  their  results  are  relevant  for  specialists  only;  others 
have  wider  relevance  also  for  patients  and  healthcare  service  users.  Overall,  however,  the  evaluation  research 
suggests  that  the  most  frequently  targeted  audiences  are  governmental  organisations,  healthcare  professionals, 
and  academia  and  researchers  (in  this  order).  These  can  sometimes  be  reached  via  publications  in  scientific 
journals (which result from some actions funded under the Health Programme), but it is important to note that 
research  is  not  the  main  focus  of  the  Health  Programme,  and  scientific  publications,  although  they  present  an 
interesting  channel  for  disseminating  information,  are  not  always  the  most  effective  way  of  reaching  directly 
those stakeholders responsible for implementing changes in the area of health. 

The  contractors  also  assessed  the  utility  of  CHAFEA’s  project  database,  which  provides  public  access  to  the 
abstracts and deliverables of co-funded actions (with the exception of tenders and direct grants to international 
organisations).  The  database  is  quite  static  and  not  always  up-to-date.  The  deliverables,  e.g. extensive  final 
project reports, are not always user-friendly and additional interactive functions are lacking that could make the 
database a useful tool providing a real service to stakeholders.  

24   http://ec.europa.eu/chafea/publications/publications_for_health_programme.html. 
25   Meetings organised in cooperation with competent Member State authorities to provide journalists and other interested audiences with 

an opportunity to learn about EU health policy and a portfolio of relevant Health Programme actions in a given topic area.  

26   http://ec.europa.eu/chafea/projects/database.html. 
27   http://ec.europa.eu/health/programme/events/ev_20120503_en.htm. 
28   http://ec.europa.eu/health/newsletter/newsletter_en.htm. 
29    http://ec.europa.eu/health/programme/policy/index_en.htm. 

19 

 
 
 
                                                      
The  ‘cluster  meetings’  organised  by  CHAFEA  in  cooperation  with  competent  Member  State  authorities  are 
assessed positively as attracting good attendance and decent press coverage. Three cluster meetings were held 
under  the  2nd  Health  Programme,  on  rare  diseases,  organ  transplantation  and  vaccination.  The  60  –  120 
participants  were  experts  in  the  relevant  field  as  well  as  journalists  from  different  EU  countries,  invited  by 
Chafea.  The  number  of  journalists  was  about  20,  covering  an  equal  number  of  Member  States.  The  average 
number of articles that appeared after the meeting in the general and specialized press is about 25 covering about 
half of the Member States, Efforts are needed to promote that articles triggered by such meetings mention the 
Programme and the EU explicitly and to widen press coverage beyond the Member State in which the meeting 
takes place.   

While the contractors could not assess the extent to which the various dissemination actions reached the various 
stakeholders,  they  recognise  that  dissemination  had  improved  over  the  course  of  the  Programme  and  that  this 
contributed  to  its  efficiency.  They  point  out  that  no  dissemination  activities  were  undertaken  specifically  to 
overcome  geographical  imbalances.  They  also  remark  that  most  publications,  guidance  documents,  etc.  are 
available only in English. 

The  smooth  functioning  of  programme  management  and  the  growing  responsibility  of  CHAFEA  across  all 
administrative functions meant that certain tasks could be streamlined and made more efficient.  

7.2.  

The Programme's effectiveness and factors that are influencing it 

Previous  evaluations  sought  to  measure  the  impact  of  the  Programme,  but  this  proved  difficult  for  reasons 
inherent  in  its  design,  the  multiplicity  of  its  actions  and  its  broad  objectives,  which  interact  with  many  other 
external factors, such as the long timescale over which effects on health materialise. 

The  majority  of  actions  in  the  framework  of  this  evaluation  were  assessed  as  successful  in  terms  of  their 
implementation, but it is not possible to ‘add up’ their outputs or to follow through on their individual impacts to 
produce a composite Programme impact (see limitations referred to in section 5.2). 

The merit of the evaluation is that it highlights numerous factors on which the Programme’s impact depends and 
which  influence  it  positively  (or  negatively  if  absent).  The  case  studies,  which  assessed  the  outcome  of  13 
actions, showed that it is of vital importance that actions:  

  have clear links to existing policy initiatives (to demonstrate how they further existing policy initiatives and 
policies; this corresponds to the ‘policy relevance’ award criterion for selecting the most relevant actions 
for co-funding); 

  have  prepared  plans  for  sustained  follow-up  efforts  (in  order  to  avoid  co-funding  actions  that  will  not 

continue once the EU co-funding is stopped); 

  work  to  propose  feasible  policy  changes  (considering  the  context)  in  the  medium  term  (this  will  help 
beneficiaries  to  concentrate  their  work  on  actions  that  can  bring  tangible  and  pragmatic  results  by 
addressing not only what has to be done, but also the challenges to be overcome and prepare the field for 
changes in the health sector); 

  have a well-delineated scope and clearly defined objectives (the absence of which may result in partners 

taking disparate action without working towards a common goal); 

  have a plausible ‘intervention logic’ (to guide the partners throughout implementation); 

 

involve  all  relevant  partners  (the  absence  of  a  strong  and  complementary  partnership  can  hamper 
implementation); 

  have strong project management (leadership is important if actions are to be implemented according to the 

plan and achieve high-quality results); 

 

involve constructive engagement from DG SANTE/CHAFEA; and 

  are implemented through the most suitable financial mechanism. 

Since ultimate responsibility for public health is (mostly) left to other organisations (in particular national health 
authorities), the success of the Programme derives from its ability to help make those other organisations (which 
range  from  international  organisations  and  national  health  ministries  to  universities  and  NGOs)  do  their  jobs 
better and more effectively. 

20 

 
 
 
Actions were more successful when they addressed identifiable policy needs, had a well-delineated scope and 
produced  results  that  could  be  readily  applied  in  practice.  While  joint  actions  and  service  contracts  met  these 
criteria to a greater extent than projects, there were a few examples among all action types where this was not 
the case. 

With the increased use of joint actions in the second half of the period, the Programme selected specific areas in 
which it especially sought to directly involve Member States authorities and the relevant bodies responsible for 
implementing health policies who have an interest in applying the Programme's outcomes, and thus maximised 
the chances of impacts materialising in the years to come. Significant achievements resulted from the majority 
of Member State health authorities being involved and cooperating very closely at the appropriate level on major 
health issues of common interest (see Annex III). 

Joint actions often are the culmination of long years of cooperation and build on previous achievements made 
possible through project grants started sometimes 10 or more years ago. Figure 8 illustrates by way of example 
the impact trajectory of the EUnetTHA joint action on HTA.  

Figure 8: 

Impact trajectory of the EUnetTHA joint action 

The challenge for the 2nd Health Programme, given its modest budget, was guaranteeing the sustainability of 
actions  and  results  of  which  the  impact  is  demonstrated  only  if  they  are  taken  up  and  used  by  Member  State 
authorities and/or other actors. This is why ‘reiterations’ of actions (possibly leading from a project to a joint 
action)  are  observed  for  some  priority  health  issues  and  no  follow-up  for  negative  priorities  that  were  not 
supported further. 

While  funding  of  recurrent  actions  was  included  as  an  option  under  the  Programme  in  order  to  meet  this 
challenge, there are two risks:  

  if funding for priorities is suspended after a certain period of time, the achievements could be lost; and  

  if the Programme spends too much on multiple iterations of a few priority actions, it could fail to identify 
meaningful new initiatives and miss opportunities to invest in new areas in rapidly changing contexts. 

Each funding mechanism has its strengths and weaknesses. Joint actions, projects and service contracts were all 
shown to be highly appropriate conditions that played to the relative strengths described in the table below. By 
contrast,  opting  for  the  wrong  funding  mechanism  in  given  circumstances  (e.g.  using  a  project  when  DG 
SANTE’s  needs  and  desired  product  are  well  defined,  which  is  better  suited  to  service  contracts)  severely 
undermined actions’ potential effectiveness (and cost-effectiveness). 

21 

 
 
 
 
 
 
 
 
 
Table 3: Conditions of success for given funding mechanisms 

Funding 
mechanism 

Joint actions 

Projects 

Ideal circumstances 

Risks / challenges  

  Clearly established case for pan-European 
collaboration at a technical (and not only 
political) level 

  Due  to  their  size  and  the  number  of 
partners  typically  involved,  joint  actions 
are  costly  to  implement  and  can  be 
difficult to manage 

  Buy-in  from  key  stakeholders  in  (nearly) 

all Member States 

 

  Feasibility  of  desired  results  already 

confirmed from previous work 

If  established  prematurely,  joint  actions 
can  be  too  unwieldy  to  provide  a  forum 
and 
for 
experimenting 

exploring 

ideas 

new 

  Political  momentum  sufficient  for  results 

to be applied in practice 

  The  chances  of  results  being  taken  up  is 
reduced  if  a  critical  mass  of  Member 
States is not secured 

  Highly  relevant  topic  but  case  for  pan-
fully 
regarding 

collaboration 

particularly 

European 
established, 
practical solutions 

not 

  Need  for  a  ‘pilot’  to  ascertain  level  of 
interest  and  feasibility  of  changing  status 
quo  

  Availability  of  strong 

leadership  and 
established  interest  from  a  smaller  group 
of committed partners to pursue a focused 
set of objectives 

  Value of collaboration beyond the level of 
to  be 

themselves  needs 

the  partners 
established 

 

If the primary focus is on networking and 
sharing  best  practices,  the  need  to  create 
more tangible results can be lost 

  Projects  often  struggle  with  national 
/ 

availability 

data 

in 

differences 
comparability  

  Overly ambitious / diverse objectives can 

reduce effectiveness  

 

If policy links are absent, it is difficult to 
overcome 
EU-wide 
implementation of results 

barriers 

for 

Service 
contracts 

  Existence  of  specific  and  clearly  defined 

  Level  of  ambition  needs  to  be  aligned 

DG SANTE needs / ideas 

with typical budgets (€100-250k). 

  Narrow  set  of  objectives  and  limited 

scope 

  Clear  link  to  specific  policy  process  or 

initiative 

  Clear  need 

for 

should  be 
established beyond interest of specific DG 
SANTE units. 

action 

  Excessive  reliance  on  service  contracts 
would be detrimental to HP inclusiveness 
(in  terms  of  types  and  geographic  spread 
of beneficiaries) 

7.4.  

Coherence and consistency with other European policies and programmes 

There  are  important  synergies  between  the  2nd  Health  Programme  and  the  Seventh  Framework  Programme 
(FP7):  actions  under  the  former  build  on  and  use  FP7-funded  research  (e.g.  on  health  threats  from 

22 

 
 
 
 
nanomaterials30) and the latter is a vehicle for the further investigation of issues and knowledge gaps that arise as 
a result of Programme actions (e.g. on specific HTA methodologies and application areas31). 

Synergy  effects  with  the  Structural  Funds  are  less  obvious,  as  the  main  results  produced  by  the  2nd  Health 
Programme  actions  such  as  networking  or  joint  solutions  (good  practice  methods  or  approaches)  do  not  lend 
themselves to implementation using co-funding from the European Regional Development Fund, the Cohesion 
Fund  or  the  European  Social  Fund.  However,  almost  €5 million  from  the  Programme  budget  was  spent  to 
promote  the  use  of  Structural  Funds  for  health.  These  actions  provided  guidance  and  awareness-raising  that 
should enable those responsible for operational programmes to address health-related issues more effectively32. 

From 2011, the Programme placed more emphasis on the Europe 2020 goals for smart and inclusive growth, by 
prioritising: 

o  actions relating to the European Innovation Partnership in the field of active and healthy ageing, which was 

set up as an Innovation Union flagship initiative; 

o  actions to address health determinants such as nutrition, smoking and alcohol abuse, which underlie many 

age-related chronic diseases; 

o  work on cancer and rare diseases; 

o  EU cooperation on HTA; 

o  work  on  the  safety  of  blood,  tissues,  cells  and  organs  (which  contributes  to  improving  health  across  the 

lifecycle, thereby contributing to healthy ageing); 

o  measures that apply information and communication technologies in the area of health; and 

o  actions aimed at bridging health inequalities to ensure better health for all and better access to healthcare 

systems.  

Comparing  spending  on  such  actions  under  the  most  relevant  priorities  in  the  first  and  second  halves  of  the 
Programme (i.e. before and after Europe 2020 was adopted), the budget for actions on active and health ageing 
increased  by  485 %  and  for  those  on  health  inequalities  by  307 %,  while  for  those  on  smart  growth-related 
priorities  it  saw  a  modest  increase  or  even  slight  decrease.  In  contrast,  funding  for  actions  addressing  health 
determinants  and  promoting healthy  lifestyles  and  those  aiming  to develop  a health  monitoring system/collect 
comparable data decreased by 17 % and 21 % respectively (Table 3). 

Table 3:   Funding for key priorities relating to Europe 2020 objectives 

Europe 
2020 
objective 

Priority 

HP funding 
2008-2010 
(€) 

HP funding 
2011-2013 
(€) 

Change 

Smart growth  Organs  and  substances  of  human  origin, 

blood and blood derivatives (1.2.2) 

4 213 499 

5 239 964 

+24 % 

Increase  healthy  life  years  and  promote 
healthy ageing (2.1.1) 

2 887 184 

16 893 162 

+485 % 

30   A series of relevant projects were funded by FP6 and FP7 which included, for example, investigations into methods for testing toxicity 
and eco-toxicity and risk assessment, and helped lay the foundation for the NANOGENOTOX Joint Action on “Safety evaluation of 
manufactured nanomaterials by characterisation of their potential genotoxic hazard”, launched under the HP in 2009. In turn,  the FP7 
project NANOREG that began in 2013 builds on NANOGENOTOX with a specific focus on regulation. 

31   Based  on  needs  expressed  by  the  resulting  EUnetHTA  network,  the  projects  ADHOPHTA,  ADVANCE_HTA,  INTEGRATE-HTA, 
and  MEDTECHTA  have  been  launched  under  FP7  on  specific  HTA  methodologies  and  application  areas,  and  there  are  annual 
coordination meetings between these and EUnetHTA. 

32   Euregio  III: Health  investments  in  Structural Funds  2000-2006:  learning lessons  to  inform  regions  in  the  2007-2013  period  (project, 
2009-2011).  This  project  evolved  significantly  over  its  lifetime  to  respond  to  emerging  issues  and  needs,  and  ended  up  generating 
evidence from existing SF projects that can show how to improve the effectiveness, efficiency and sustainability of direct health system 
investments using SF in the next cycle (2014-2020). The results have been relatively widely used, inter alia for training sessions and a 
guide  for desk  officers  in  or  with  DGs  REGIO and  EMPL,  and  as  input  for the  discussions  of  subgroup  2 of  the  Council  reflection 
process on health systems. 

23 

 
 
 
 
                                                      
Address  health  determinants  and  promote 
healthy lifestyles (2.2.1) 

Prevention  of  major  diseases  of  particular 
significance, and rare diseases (2.2.2) 

Develop  a  sustainable  health  monitoring 
system and collect comparable data (3.2.1) 

32 897 669 

27 221 835 

-17 % 

19 103 140 

19 920 192 

+4 % 

15 719 845 

12 490 561 

-21 % 

Inclusive 
growth 

Identify  the  causes  of,  address  and  reduce 
health inequalities (2.1.2) 

7.    CONCLUSIONS 

3 552 153 

14 440 968 

+307 % 
Source: CHAFEA database and DG SANTE 

The main conclusions that can be supported based on the evaluation can be summarised as follows:  

  The  management  of  the  2nd  Health  Programme  improved  compared  with  the  1st  Public  Health 
Programme  as  a  result  of  being  entrusted  to  the  Executive  Agency  and  of  the  introduction  of  different 
financial  mechanisms,  such  as  joint  actions,  operating  grants  and  direct  grants  with  international 
organisations, which better serve stakeholders’ needs and respond to their expectations.  

By following up many recommendations from the mid-term evaluation, implementation improved further 
in  the  second half of  the  Programme,  through  the  use of more  strategic  programming,  the  systematic 
use  of  EU-added-value  criteria  in  grant  applications  and  selection,  clearer  guidance  for  applicants 
and  better  contact  with  applicants  and  beneficiaries.  However,  there  are  still  concerns  around 
monitoring,  dissemination  of  results  and  administrative  burden  for  applicants  and  beneficiaries.  Good 
monitoring  is  crucial,  also  for  communicating  Programme  results,  and  increased  systematic  efforts  are 
needed  to  ensure  dissemination  with  clear  strategic  objectives,  targeting  stakeholders  with  the  most 
influence and power.  

  The  Programme  was  highly  coherent  with  the  Europe  2020  strategy  for  smart  and  inclusive  growth. 
Since  2011,  its  policy  relevance  was  enhanced  by  the  use  of  more  joint  actions  and  tenders,  and  more 
funding  for  actions  that  focus  on  promoting  healthy  and  active  ageing  and  reducing  health  inequalities. 
However,  every  case  of  relevant  need  does  not  necessarily  imply  a  strong  case  for  EU  involvement.  To 
maximise  the  impact  of  the  Programme,  the  relevant  actions  should  demonstrate  clearly  their  EU 
added value. This is why criteria for EU added value were established in the course of the Programme and 
applied in the assessment of proposals. For the 3rd Health Programme, the Commission went a step further 
by  proposing  that  Regulation  (EU)  No 282/2014  contained  an  explicit  reference  to  these  criteria,  which 
continue  to  be  integrated  in  the  evaluation  process  for  awarding  funding  to  ensure  that  all  co-funded 
actions deliver EU added value.   

In comparison with other financial mechanisms, joint actions often generate substantial added value, in 
some  cases  leading  to  tangible  cost  savings  in  addition  to  providing  useful  lessons  for  the  Member 
States involved. They deal with priorities determined through a comprehensive appraisal of public-health 
needs in Europe, with input from Member States and other interested parties, e.g. expert committees. They 
often support a policy process in a specific field of health33 and encourage Member States to cooperate in 
the implementation of an existing legal framework.34 Some joint actions address emerging health problems 
in the global health environment; pathogens do not respect borders and can affect several Member States, 
so common action is often required.35 Joint actions can also arise in response to ‘horizon-scanning’ work 
by EU expert committees to identify emerging health problems in Europe which could become a priority 
for action at European level.36  

The  ‘project’  funding  mechanism,  as  used  for  the  majority  of  actions  aimed  at  health  promotion  and 
accounting for 57 % of the total funding awarded, responds to EU-added-value criteria relating mainly to 
best  practices,  benchmarking  for  better  decision-making  and  networking.  These  criteria  have  weak 
links  to  tangible  policy  benefits  and  need  to  be  more  clearly  defined  and  communicated  so  as  to  help 
applicants  to  submit  proposals  and  ensure  that  actions  deliver  more  tangible  and  concrete  benefits. 
Therefore, the evaluators’ suggestion for re-working these three criteria seems to make sense. This could 

33   e.g. the EJA joint action; http://ec.europa.eu/chafea/projects/database.html?prjno=20112201. 
34   e.g. the FOEDUS joint action; http://ec.europa.eu/chafea/projects/database.html?prjno=20122101.  
35   e.g. the QUANDHIP joint action; http://ec.europa.eu/chafea/projects/database.html?prjno=20102102. 
36   e.g.  the  EQUITY  ACTION  and  ECHIM  joint  actions;  http://ec.europa.eu/chafea/projects/database.html?prjno=20102203  and 

http://ec.europa.eu/chafea/projects/database.html?prjno=20082391. 

24 

 
 
 
                                                      
be very useful for credibly demonstrating how such actions lead to more concrete benefits over the longer 
term.  

  The Programme finances actions with the potential to influence health policies positively at national and 
EU  level,  taking  into  account  certain  conditions,  including  long-term  financing  for  secured 
sustainability  of  activities  and  outcomes.  Nevertheless,  the  contractors  found  it  difficult  to  assess  the 
impact  of  the  Programme  as  a  whole,  given  its  broad  objectives,  the  multiplicity  of  priorities  and  the 
absence  of  indicators  for  measuring  progress  (an  issue  raised  in  earlier  evaluations).  Recommendations 
from the mid-term evaluation calling for a focus on a restricted number of actions with defined progress 
indicators linked to the corresponding Programme objectives have been implemented in the design of the 
3rd Health Programme and will be assessed in its mid-term evaluation in 2016-2017. 

  Regarding the low participation of EU 12 Member States in the Health Programme, the evaluation findings 
confirm that efforts should be continued to encourage greater participation from underrepresented Member 
States, inter alia by targeting key governmental institutions and drawing attention to the opportunities on 
offer.  If  it  continues,  the  low  participation  of  some  Member  States  may  hamper  the  success  of  the  3rd 
Health Programme.  

Going forward, and given that public-health capacity-building is explicitly included in the legal basis of the 
3rd Health Programme, it would seem necessary for the concept to be clearly defined, agreed, factored into 
clear  Programme  priority-setting  and  the  conducive  design  of  individual  actions  (including  possible 
specific  mechanisms  to  support  ‘weaker’  Member  States),  and  further  elaborated  in  monitoring 
arrangements and future evaluations.  

Also, wider dissemination of outputs and results can provide good examples and convincing arguments for 
more involvement and leadership for all Member States/participating countries interested in the Programme.  

25 

 
 
 
 
 
ANNEX I – EVALUATION MATRIX 

Evaluation questions 

Judgment criteria 

Preliminary indicators 

Data sources 

Evaluation bloc (a) Programme management tools 

i. To  what extent  have the recommendations of 
the  mid-term  evaluation 
the 
management  and  the  design  of  the  Programme 
been implemented? 

concerning 

  Extent to which a strategic multi-annual 
planning with long-term targets was 
developed.  

  Extent to which the Programme continues to 
employ the current system of different and 
diversified financial mechanisms.   

  Extent to which clear guidelines are provided 
to potential applicants and applicants.  

  Extent to which EU added value is being 

emphasised, particularly at the proposal stage.  

  Extent to which Programme application 
procedures have been simplified and 
rationalised. 

  More tangible and focussed objectives (SMART) 

  Desk research (including statistical 

developed 

review and analysis) 

  NFP survey  

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

-  External stakeholders 

  Evidence of long-term planning  
  Targets set in consultation with national health experts 
  Indicators developed 
  Number of actions funded (2008-13) and budget allocated 

by financial mechanisms 

  Officials’ and stakeholders’ views on the balance of funding 

mechanisms 

  Evidence of clear guidelines provided to potential 

applicants and applicants (e.g. on intervention logics, 
ToCs, SMART objectives, indicators, definitions of certain 
terms etc.)_ 

  Evidence of retention of  requirement for proposals to 

outline compliance with AWP priority areas and HP 
objectives 

  Stakeholders’ perceptions on whether clear / simple 

guidelines are provided 

  Existence and level of dissemination of a nomenclature for 

explaining EU added value etc. 

  Level of utilisation of this nomenclature in proposals and 

actions 

  Evidence of simplified and rationalised Programme 

application procedures (including “lighter” application 
process, use of framework contracts to reduce repetitive 
and burdensome tendering procedures) 

  Officials’’ and stakeholders’ perceptions on whether 

Programme application procedures have been simplified 
and rationalised and what more could be done 

   

  Extent to which monitoring has been enhanced    Evidence that applicant satisfaction surveys are still being 

undertaken 

  Evidence of the monitoring of types of organisations 

applying for funding 

  Evidence of more in depth assessments of samples of 

actions 

26 

 
 
 
 
 
 
 
 
 
Evaluation questions 

Judgment criteria 

  Extent to which actions and their results have 

been better disseminated 

ii.  How  effective  have  recent  changes  in  the 
emphasis  on  and  use  of  specific 
funding 
mechanisms  (i.e.  use  of  Joint  Actions,  balance 
between calls for proposals and calls for tender) 
been  in  delivering  policy-related  outputs,  and 
the  geographical 
impact  on 
the 
what  was 
distribution of beneficiaries? 

influence 

recommendations 

iii.  To  what  extent  did  the  implementation  of 
previous 
the 
Programme’s  other  operations,  including  the 
recruitment  of  beneficiaries  and  the  level  of 
participation of all Member States in Programme 
actions  (including  the  facilitation  of  participation 
from low GNI countries)? 

  Extent to which better use has been made of 

synergies at national, European and 
international levels 

  Extent to which changes in the emphasis on 
and use of specific funding mechanisms 
have contributed to policy-related outputs 

  Extent to which changes in the emphasis on 
and use of specific funding mechanisms has 
had any impact on the geographical 
distribution of beneficiaries 

  Extent to which the implementation of 

previous recommendations has impacted on 
the recruitment  of beneficiaries 

  Extent to which the implementation of 

previous recommendations has impacted on 
the level of participation of all Member 
States, and in particular has facilitated 
participation from low GNI countries 

Preliminary indicators 
  Evidence of dedicated budgets for dissemination at action 

Data sources 

level 

  Evidence of clearer guidelines on target groups, 

dissemination plans 

  Evidence of increased dissemination activities from DG 

SANCO / action leaders to stakeholders (e.g. Programme 
Committee, EP, Council, CoR, wider audiences) 
  Evidence of (increased) collaboration with relevant 

interested parties/organisations, e.g. through meetings 
and conferences 

  Evidence of increased data sharing between HP actions 
and similar activities at national, European, international 
levels 

  Evidence and examples of policy-related outputs 
  Evidence demonstrating links between  the emphasis / use 
of specific funding mechanisms and policy-related outputs 

  Officials’ and stakeholders’ perceptions on whether 

changes in the emphasis on and use of specific funding 
mechanisms have contributed to policy-related outputs 
  Evidence and examples of changes in the geographical 

distribution of beneficiaries 

  Evidence / examples demonstrating links between  the 
emphasis / use of specific funding mechanisms and 
changes in the geographical distribution of beneficiaries 

  Officials’ and stakeholders’ perceptions on whether 

changes in the emphasis on and use of specific funding 
mechanisms have contributed to changes in the 
geographical distribution of beneficiaries 

  Desk research  

(including  statistical 
analysis) 

review  and 

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

-  External stakeholders 

  Quantity of proposals received per year under the 

  Desk research (including statistical 

different funding mechanisms 

review and analysis) 

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

  Evolution of the profile of applicants (by Member State 

and by partner-type) 

  Officials’ and stakeholders’ perceptions on whether and in 

which ways the implementation of previous 
recommendations has impacted on the recruitment of 
beneficiaries 

  Evolution of the level of participation of Member States 
  Officials’ and stakeholders’ perceptions on whether and in 

which ways the implementation of previous 
recommendations has impacted on the level of 
participation of all Member States 

27 

 
 
 
 
 
 
 
 
Evaluation questions 

Judgment criteria 

iv. What are the state of the art tools in terms of 
monitoring project outputs that could be applied 
to  the  Programme,  what  are  the  expected 
benefits  against  costs  and  how  could  they  be 
implemented? 

  Extent to which there are relevant state of the 
art tools for monitoring project outputs that 
could be applied to the Programme 

  Benefits against costs of applying tools for 

monitoring project outputs  

  Ways in which these tools could be 

implemented to the Programme, including 
associated risks 

Preliminary indicators 
 

Inventory of state of the art tools for monitoring project 
outputs  

Data sources 
  Desk research 

  Examples from national and EU Programmes 

  NFP survey 

  SWOT  analysis of identified tools / monitoring systems 
  Officials’ and stakeholders’ perceptions on benefits and 

 

costs of applying identified tools for monitoring project 
outputs 

  Officials’ and stakeholders’ perceptions on ways in which 
these tools could be implemented to the Programme, and 
associated risks 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 
-  Representatives of other 

programmes and initiatives (e.g. 
National Institutes of Health, 
National Institute for Health 
Research) 

Evaluation bloc (b) Programme dissemination practices 

v.  

  Extent to which individual HP actions develop 

  % of actions that developed explicit dissemination plans 

  Review of database / documentation 

a)  To  what  extent  have  the  actions/  outcomes/ 
results  of  the  Second  Health  Programme  been 
published?  To  what  extent  are  they  (made) 
accessible  to  the  international  scientific  and 
health  community,  to  health  policy  makers,  civil 
society, and to the wider public in the EU?  

b) Are the results published and disseminated in 
a sustainable way?  

c)  How  useful  is  the  EAHC  database  in  this 
context? How can it be improved?  

d)  Which  other  tools  would  be  useful  in  this 
context? 

(for  part  d)  of  the  question,  see  question  viii. 
below) 

dissemination strategies 

  Extent to which relevant stakeholder groups 

are targeted (as recipients or multipliers of 
information) 

  Extent to which dissemination strategies for 

action results are implemented effectively 

  Breakdown of stakeholders targeted 

  Funds budgeted / spent on dissemination (as a share of 

on 90 actions 

  Case studies 

total action budget) 

  Stakeholder analysis 

  Access (no. of hits) to selected actions’ websites 

  Stakeholder interviews 

  No. of copies of final report sent / distributed (electronic 

and hard copies) 

  No. of dissemination events organised (conferences etc.) 

to present results 

  Use of other dissemination tools (e.g. newsletters) 

  Evidence of uptake / implementation of results, 
depending on dissemination strategies pursued 

  Effectiveness of dissemination strategies / tools / 

activities / target audiences as perceived by stakeholders 

  Quantity of scientific output stemming from 

  Number of articles published in peer-reviewed journals 

  Bibliometric analysis of selected 

Programme actions 

  Quality of scientific output stemming from 

Programme actions 

  Number of citations of such articles 

 

Journal impact factor  of HP2-related publications 

  H-index (citation index ) of HP2-related publications 

actions 

28 

 
 
 
 
 
 
 
Evaluation questions 

Judgment criteria 

Preliminary indicators 

Data sources 

  Sustainability of dissemination efforts 

  Extent to which further dissemination is foreseen / carried 

  Review of database / documentation 

is 

the 

relation  between 

the 
vi.  What 
publications/activity  reporting  and  the  Member 
State  participation 
the  Second  Health 
in 
Programme,  the  number  of  health  scientists, 
public  health  specialists  and  physicians  per 
Member States? Are patterns identifiable? Have 
dissemination activities been undertaken in way 
to  overcome  possible  geographical  imbalances 
in certain actions? 

out after the end of HP2 funding 

on 90 actions 

  Extent to which relevant websites remain live / updated 

  Case studies 

following action completion 

  Extent to which scientific output remains accessible / is 

cited (see above) 

  Usefulness of EU (DG SANTE / CHAFEA) 
dissemination activities and tools, in 
particular the database 

  No. of visits to CHAFEA database 

  Review of database and other tools / 

  Usefulness of database as perceived by stakeholders 

  Stakeholder views and perceptions of other tools (events, 

activities 

  Stakeholder interviews 

publications, etc.) by DG SANTE / CHAFEA 

  NFP survey 

  Extent to which participation / publication 

  Participation of organisations from different MS in actions 

  HP action ‘portfolio’ review 

rates by MS reflect the relative public health 
capacity of MS 

(as coordinator or collaborator), per year 

  Data review from other sources 

  No. of scientific publications related to HP actions, by MS  

(ECHI, other sources) 

  Extent to which dissemination has 
contributed to reducing imbalances 

  Measures of public health capacity: (1) health resources 

(financial / human), (2) health research (GERD, 
publications, clinical trials) 

  Relationship between the above, patterns and trends 

over time 

  No. and type of measures adopted to overcome 

  Stakeholder interviews 

imbalances 

  Extent of change that can be attributed to such measures 

  No. of dissemination activities undertaken to redress 
geographical imbalances in selected actions   

  Geographical differences in the level of satisfaction of 

stakeholders of selected projects with the dissemination 
efforts made 

  NFP survey 

  Case studies 

governments 

vii.  To  what  extent  do  stakeholders  other  than 
Member  State 
(subnational 
regional  organisations,  civil  society,  social 
partners  etc.)  promote  Programme  outcomes 
and  results,  and  via  which  channels?  This 
should  consider  both  organisations  funded  by 
the programme, and others. 

  Extent and outreach of promotion activities 
carried out by the various non-State actors  

  Extent of recourse to the different 

communication channels (web, other media, 
conference /meetings etc.) 

  Tangible effects on health policy / strategy at 
the various level related to dissemination 

  % of actions with non-State actors promoting 

  Review of database / documentation 

outcomes/results spontaneously (broken down by type) 

on 90 actions 

  Frequency of utilisation of the various communication 

channels for the above activities  

 

Level of outreach of the above dissemination actions (for 
selected projects) 

  Online questionnaire with National 

 

Focal Points 
Interviews with non-governmental 
promoters / disseminators 

29 

 
 
 
 
Evaluation questions 

Judgment criteria 

Preliminary indicators 

  No. of reported cases of tangible effects (and type) on 
policy / strategy from the above activities (selected 
actions) 

Data sources 

  Case studies (focused on 

dissemination) 

the  current  dissemination 
viii.  How  could 
practices  be  improved  to  increase  return  on 
investment? 

  Room for improvement of current 

  Overview of other relevant possible dissemination tools 

 

Ideas gleaned from interviews / 
surveys / other sources 

dissemination practices 

  Cost-effectiveness of other dissemination 
tools / approaches that could be used 

(to be identified) 

  Estimated costs of alternative approaches 

  Cost-benefit / cost-effectiveness 

 

Likely benefits 

analysis 

Evaluation bloc (c) Impact of the programme  

ix:  How  and  to  what  extent  has  the  Second 
Health  Programme  supported  Member  States’ 
health  policy  and  actions  (in  relation  to  the 
provisions 
and 
cooperation 
coordination in Article 168 of the Treaty)? 

support, 

on 

  Extent to which the Programme has 

supported Member States’ health policy and 
actions 

  Ways in which the Programme has supported 
Member States’ health policy and actions 

  Evidence and examples of ways in which the Programme 
has supported Member States’ health policy and actions 

  Extent to which the Programme meets the essential 

criteria for maximising its potential impact 
  Stakeholders’ perceptions on ways in which the 

  Desk research - screening exercise 

and in-depth review 

  Case studies 

Programme has supported Member States’ health policy 
and actions 

  NFP survey 

x:  Which  are  the  main  health  policy  areas  in 
which  progress  has  been  achieved  due  to  the 
support  of  the  Health  Programme,  and  what 
constitutes this progress? 

 

Identification of the main health policy areas 
in which progress has been achieved as a 
result of Programme support 

 

Identification of actions / results illustrating 
the progress achieved 

  Stakeholders’ perceptions on (i) whether Programme 
activities are additional to what MS would have done 
anyway, and (ii) what might have happened in the 
absence of the Programme 

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

-  External stakeholders 

  Evolution over time in the main health policy areas where 

  Desk research  -  screening 

actions have been funded 

  Evidence and examples of health policy areas where 
significant progress has been achieved due to the 
support of the Programme based on an assessment in 
relation to the 7 added value criteria 

  Stakeholders’ views on the health policy areas where the 
Programme has had the greatest impact and where it 
should be providing more support 

  Stakeholders’ perceptions on (i) whether Programme 
activities are additional to what MS would have done 
anyway, and (ii) what might have happened in the 
absence of the Programme 

  Degree to which given actions are sustainable, i.e. would 
continue with a reduction in / in the absence of EU 
funding 

  Extent to which given actions meet the pre-defined 

exercise, in-depth review and added 
value analysis 

  Case studies 

  NFP survey 

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 
and National Focal Points 

-  External stakeholders 

30 

 
 
 
 
 
 
 
 
 
 
Evaluation questions 

Judgment criteria 

Preliminary indicators 

Data sources 

essential conditions for actions (i.e. the action has 
defined SMART objectives and targets; the action has 
defined a clear target group; the action has developed 
clear implementation and dissemination strategies; the 
action has delivered its main deliverables to a sufficient 
standard; the action has covered its stated target group; 
the action has met its stated targets; the action has 
disseminated its outputs / results to plan) 
  Stakeholders’ perceptions on the actions / results 

 

illustrating the progress achieved 
In-depth review of Programme impact for a selection of 
actions and analysis of when this can be measured 
  Stakeholders’ perceptions on time-adapted ways in 

which to measure the Programme’s impact 

xi: What are reasonable assumptions on the way 
to  measure  the  impact  of  the  programme  in 
terms of timelines a) short-term, b) middle-term, 
c) long-term and d) in relation to average project 
trajectories? 

 

Identification of assumptions to measure the 
impact of the programme in terms of 
timelines a) short-term, b) middle-term, c) 
long-term and d) in relation to average 
project trajectories 

xii:  Which  factors/reasons  may  intervene  and 
influence  positively  or  negatively  the  impact  of 
the Programme? 

 

Identification of factors/ reasons that can 
positively or negatively influence the impact 
of the Programme 

 

Identification of examples of  factors / reasons that can 
positively or negatively influence the impact of the 
Programme 

  Stakeholders’ perceptions on factors / reasons that can 
positively or negatively influence the Programme 

xiii:  What  are  the  main  lessons  than  can  be 
drawn to ensure an overall successful transition 
from the second to the third Health Programme? 

 

Identification of lessons to ensure an overall 
successful transition from the second to the 
third Health Programme 

  Evidence and examples of lessons to ensure an overall 
successful transition from the second to the third Health 
Programme 

  Case studies 

  NFP survey 

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

-   

  Case studies 

  NFP survey 

 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

-  External  stakeholders 

  Desk research - Desk review and 

analysis of relevant documentation 

  Stakeholders’ perceptions on lessons to ensure an 

  Desk research  -  screening 

overall successful transition from the second to the third 
Health Programme 

exercise, in-depth review and added 
value analysis 

  Case studies 

  NFP survey 

31 

 
 
 
 
  
 
  
 
 
 
  
Evaluation questions 

Judgment criteria 

Preliminary indicators 

Data sources 
 

Interviews with:  
-  DG SANTE and CHAFEA officials 
-  Programme Committee members 

and National Focal Points 

-  External stakeholders 

Evaluation bloc (d) Synergies with other policies and programmes  

xiv. What synergies are there with other policies 
and programmes of the Commission such as the 
European  Structural  and  Cohesion  Funds,  the 
programmes  managed  by  DG  RTD,  other  DGs 
(in  particular  EMPL,  CONNECT)  and  to  what 
extent  did  the  Health  Programme  underpin  the 
-focus  on 
Commission’s  general  objectives 
Europe  2020  and  their  objectives  related  to 
social  policy  (e.g.  the  renewed  Social  Agenda) 
and  economic  growth  (research  and  innovation, 
competitiveness)? 

  Extent to which health interventions funded 

  Comparison of health-related funding (HP, FP7, ERDF, 

  Desk research (including value 

under the HP and key other programmes 
have potential synergies (theoretical 
analysis) 

  Extent of actual synergies (illustrated by 

 

examples) 
Level of coordination at planning and 
implementation level (delivery)  

ESF) 

  Specific examples of synergy effects (if any) 

  Existence of EU-level mechanisms for coordination of 
priorities, exchange of results and/or cross-fertilisation 

  Satisfaction of key staff with the degree of synergies / 

complementarity     

 

chain analysis) 
Interviews with officials representing 
other services and programmes 

  Coherence of HP with main EU policy 

  No. of HP2 projects consistent with Europe 2020 

  Desk research 

objectives (Europe 2020) 

objectives 

  Effects of HP2 actions on smart / inclusive growth 

32 

 
 
 
 
 
ANNEX II – RECOMMENDATIONS APPLIED 

Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

CONCEPTION 

More tangible and focussed objectives 
(SMART37); better defined strategic 
framework for the HP and the 
development of indicators  

Intended effect(s) 

Preliminary assessment of implementation 

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

To what extent were indicators developed at a 
strategic level (rather than action level) under the 2nd 
HP to show progress / achievements relative to the 
stated objectives? 

To ensure the programme is focussed on 
certain public health issues (especially 
those that are difficult for MS to reach 
individually) and for targets / indicators to 
be developed to measure the extent to 
which these objectives / priority areas are 
achieved as well as greater transparency 
in how priorities are reached in Annual 
Work Programmes 

2nd HP: 

2012 and 2013 AWPs focused on Europe 2020 
relevance 

Evidence of shift from ‘bottom up’ development 
process to ‘strategic’ prioritisation (based on 
increased input from DG SANTE senior leaders) 

(also see “Design” Recommendation – to 
give EU added value greater prominence 
in proposal stage) 

3rd HP: 

Objectives re-defined for the 3rd HP38 with the 
result that they are much more focused on 
supporting EU priorities and therefore the pursuit 
of EU added value 

Progress indicators have been introduced for 3rd 
HP39 (however there is no reference to them in 
official documents consulted to date, e.g. the 
AWP 2014) 

37   Specific, Measurable, Attainable, Relevant, Time-bound = SMART 
38   Promote health, prevent diseases and foster supportive environments for healthy lifestyles taking into account the 'health in all policies' principle; Protect Union citizens from serious cross-border health threats; 

Contribute to innovative, efficient and sustainable health systems; Facilitate access to better and safer healthcare for Union citizens (see Programme Regulation 282/2014/EC. URL: http://eur-
lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014R0282&from=EN) 

39   See presentation of the third health programme (see http://ec.europa.eu/health/programme/docs/ev_20141104_co01_en.pdf)  

33 

 
 
 
  
                                                      
Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Develop long-term planning and targets 
in consultation with national health 
experts 

Intended effect(s) 

Preliminary assessment of implementation 

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

To facilitate:  

  Setting appropriate priority 

actions vis-à-vis health needs 
and objectives in individual 
Member States  

  Selecting financing 
mechanisms  

To ensure an appropriate spread among 
the objectives and priorities. 

Greater share of funding allocated to joint 
actions, service contracts and operating grants 
(cf. smaller share to presidential conferences, 
projects) from 2011 – 2013 cf. 2008-2010. 

Assess in greater detail whether DG SANTE explain 
and document this process clearly and whether the 
AWPs provide sufficient rationale / justification behind 
varying levels of funding for each objective 

More emphasis (both in terms of number of 
actions and budget spent) on “Health Promotion” 
cf. “Health Security” and “Health Information”.  

Has value for money improved through greater use of 
JAs?  

It remains to be seen (through survey analysis 
and interviews) whether divergences in the focus 
of funding by strand can be attributed to differing 
health needs in each country 

According to our preliminary assessment, the 
Annual Work Plans do not provide a clear 
rationale / justification behind varying levels of 
funding for each objective  

All countries (with the exception of 
Luxembourg) received more funding for 
Health Promotion than Health Security or 
Health Information40, although there were 
some exceptions41 

DESIGN 

Retain current funding mechanisms 
and consultation of action leaders on 
their experience with new funding 
mechanisms 

To continue to provide insight into the 
pros and cons of each funding 
mechanism, what aspects they would 
change / improve at the end of each 
action 

All funding mechanisms continue to be used but 
as described above, the relative proportion of 
budget allocated to each funding mechanism 
shifted following on from the Mid-term evaluation 

Look into degree of consultation with action leaders 
once an action has come to a close 

40   Note that country-level data is only available for CHAFEA managed actions, and not for DG-SANTE managed actions. 

41   All countries (with the exception of Luxembourg) received more funding for Health Promotion than Health Security or Health Information. The proportion ranged from 29% of funds received (Luxembourg) to 
90% (Iceland). For Health Information and Health Promotion the lower range was sometimes very low: for example 5% or less of  funds went to Health Information in Iceland, Cyprus, Bulgaria and Romania, 
while 10% or less of funds went to Health Security in Iceland and Finland.  

34 

 
 
 
                                                      
Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Retain requirement for proposals to 
outline compliance with AWP priority 
areas and HP objectives  

Intended effect(s) 

Preliminary assessment of implementation 

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

DG SANTE continue assessing proposals 
according to their policy relevance 

External evaluators continue rating 
proposals according to their evidence 
base42 

Guides include information on awarding criteria 
which make reference to policy and contextual 
relevance of action as well as reference to 
evidence base 

Emphasise and clarify EU added value 
particularly in the proposal/application 
process stage 

The promotion of clearer EU AV allows 
the programme to better achieve its 
objective  

Clarity /applicability of EU AV (in general and for 
applicants)? 

All EU AV criteria are still not altogether clear  

Poster on EU added value produced for High 
Level Conference on Health Programmes43  

EU Added Value is included in the application 
guides but the information has not evolved since 
the mid-term evaluation / does not make 
reference to clear guidelines on EU added value 

Meaning of EU added value is outlined in FAQ for 
Health Programme calls for proposals 2013 on 
the CHAFEA website (not clear when this was 
first included)44 

3rd HP:  

EU AV criteria are detailed in Programme 
Regulation 282/2014/EC (6)45  

42   For example, the 2011 Guide for Operating Grants asks for information on the “evidence base (citing data, studies, etc.) for the general activities and overall objective of the organization / network.” 
43   http://ec.europa.eu/health/programme/docs/ev_20120503_eu_added_value_en.pdf  
44  
45  

http://ec.europa.eu/chafea/health/faq.html  
“The Programme should be a means of promoting actions in areas where there is a Union added value that can be demonstrated on the basis of the following: exchanging good practices between Member States; 
supporting networks for knowledge sharing or mutual learning; addressing cross-border threats to reduce their risks and mitigate their consequences; addressing certain issues relating to the internal market where 

35 

 
 
 
 
                                                      
Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Intended effect(s) 

Preliminary assessment of implementation 

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

MANAGEMENT 

Monitor the organisations applying for 
funding and carry out a more in depth 
assessment of sample of actions  

To ensure there is equal access for all 
applicants to receive funding  

To provide insight into the actions funded 
but also to have data available to 
disseminate to interested parties  

Reference is made to EU AV in the (annexes to) 
AWP 201446 

factsheet  on  EU  AV 

Calls for proposals for actions make reference to 
EU  AV47  and 
to  be 
distributed along with calls for proposal from 2014 
onwards  as  well  as  a  reference  included  in  the 
guides for applicants48  

Info  days  for  potential  applicants  organised 
on 11 April and 11 June 2014 

Annual consensus reports produced for given 
projects that have ended by CHAFEA 
following evaluations by three experts. Focus 
placed in particular on projects where 
problems have been identified.  

Overall, government organisations make up 
the largest proportion of beneficiaries, 
followed by non-profit organisations and 
academic organisations. A very small 

Explore the make-up of main and associated 
beneficiaries across the EU by action type and strand 

the Union has substantial legitimacy to ensure high-quality solutions across Member States; unlocking the potential of innovation in health; actions that could lead to a system for benchmarking to allow informed 
decision-making at Union level; improving efficiency by avoiding a waste of resources due to duplication and optimising the use of financial resources.”  

46  

http://ec.europa.eu/health/programme/events/adoption_workplan_2014_en.htm  

47   See for example, the factsheet for Project grants: http://ec.europa.eu/chafea/documents/health/factsheets_hp_pg_en.pdf  

48   See 

for 

example 

via: 
http://ec.europa.eu/chafea/health/index.html  ;  project  factsheets:  http://ec.europa.eu/chafea/health/projects.html;    operating  grant  factsheet:  http://ec.europa.eu/chafea/health/grants.html;  joint  action  factsheet: 
http://ec.europa.eu/chafea/health/actions.html  

http://ec.europa.eu/research/participants/data/ref/other_eu_prog/hp/hp-call-guide-app-pj_en.pdf 

applicants: 

factsheets 

Project 

guide 

2014 

and 

EU 

the 

for 

36 

 
 
 
 
                                                                                                                                                                                                                                                               
Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Intended effect(s) 

Preliminary assessment of implementation 

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

Provide better guidance for proposals 
and simplification of application 
processes  

More, better applications leading to 
actions with more relevant, tangible 
results 

proportion of partners are classified as 
commercial organisations, while the 
proportion of non-governmental 
organisations and international public 
organisations and private individuals is 
negligible. 

The in-depth review provided more detailed 
information on sub-groups of beneficiaries, 
for example, more or less a quarter of 
government organisations are “policy makers 
and regulators” (28%), “general public health 
organisations/institutions” (28%), 
“Specialised health organisations and 
institutes” (23%) while the remaining share 
were divided between “Healthcare providers 
and commissioners” (14%) and “other” (i.e. 
non-health) government bodies (7%).   

2nd HP:  

Guidance document for actions developed by 
CHAFEA and guide available for each funding 
mechanism together with call for proposals 

Positive feedback from survey of applicants to 
calls for proposals 2008 – 2013 regarding both 
the guidance documents and helpdesk services 
but still room for improvement (for example 
Frequently Asked Questions section should 

37 

Identify ways to manage and implement on-going 
improvements (e.g. simplifying technical and financial 
parts of application forms) using survey results and 
feedback  

Look into benefits of NFPs 

Test sub-recommendations outlined in the Mid Term 
Evaluation which relate to improving guidance for 

 
 
 
Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Intended effect(s) 

Preliminary assessment of implementation 

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

contain more technical answers rather than 
mainly general ones) 

applicants:  

From evaluators point of view, the administrative 
side of applications could be improved and 
simplified (suggestions include move to electronic 
submissions, improvements to the structure of 
information provided) 

3rd HP: 

Simplification measures introduced, including the 
electronic submission of proposals via the 
Electronic Exchange System of the EC 
Participant Portal49, new procedures for granting 
Joint Actions (i.e. without the need for a call for 
proposal)  and operating grants, among others.   

 

 

 

 

 

 

intervention logics and theories of change 
to participants50; 

setting indicators that could provide an 
insight into the extent to which the 
outcomes are being / have been achieved. 
Without these it is difficult to determine how 
effective an action has been and the extent 
of its impact at the point of assessment; 

how to set SMART objectives in order to 
effectively measure progress; 

definitions of what is required in certain 
sections of the application form, i.e. 
“evidence base”, given that applicants 
might have different understandings of 
certain terms used (without interfering in 
the peer review process and without 
encroaching on the capacity of the 
applicants to formulate the evidence base);  

assessing potential “EU added value” along 
clear and quantifiable criteria  

defining target groups / dissemination plans 
/ evaluation plans. 

49   http://ec.europa.eu/chafea/news/news327.html  
50  

definitions and very clear examples of Inputs, Outputs, Results, Outcomes and Impacts of an action 

38 

 
 
 
                                                      
Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Ensure better dissemination of actions 
and their results, including dedicated 
budgets, by DG SANTE / action leaders 
to stakeholders (e.g. Programme 
Committee, EP, Council, CoR, wider 
audiences) 

Intended effect(s) 

Preliminary assessment of implementation 

The better and more systematic the 
dissemination, the wider the reach of the 
programme 

DG SANTE: 

Increased focus on dissemination of results by 
DG SANTE (e.g. brochures, covering broad 
range of actions, grouped under thematic areas)   

Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

Where is there still room for improvement in terms of 
dissemination, e.g. on the website/newsletters, 
production of short summaries for national policy 
makers? The interviews and case studies will 
inform this. 

The Mid-term evaluation mentioned a planned 
conference (High Level Conference on EU Health 
Programmes: results and future perspectives”) in 
particular as having significant potential to 
support dissemination efforts51 

Which dissemination methods are most effective 
(level of dissemination / dissemination method)? 
Further research to be carried out on this as part 
of the analysis of the results of the in-depth 
review. 

CHAFEA:  

Improvements made to CHAFEA database of 
actions 

Make use of synergies 

Multiplying the impact of the HP through 
disseminating results and collaborating 
with relevant interested 

2nd HP: 

Roadmap for dissemination is currently being 
designed and is in draft format. 

51   http://ec.europa.eu/health/programme/events/ev_20120503_en.htm 

39 

Use the case studies to assess: Whether target 
audiences identified? Are target audiences identified 
sufficiently specific? Do actions have clear 
dissemination plans? Do they allocate a budget to 
dissemination in their proposals? 

Has communication between DG SANTE, CHAFEA 
and the Programme Committee improved (for 
example so that the later are more aware of 
dissemination efforts)? To be explored during the 
interviews. 

To what extent were synergies capitalised on under 
the 2nd HP? (One of the research questions for the 
case studies – that related to EU added value - 
will be used to hone in on the extent to which 

 
 
 
 
                                                      
 
Issues tested further  during the ex-post final 
evaluation of the HP 2008-2013 

potential synergies and linkages were capitalised 
on during the 2nd HP)  

Recommendation(s) of the mid-term 
evaluation of the HP 2008-2013 

Intended effect(s) 

Preliminary assessment of implementation 

parties/organisations 

The activities (including a specific framework 
contract) will disseminate the outputs and 
outcomes of the finished actions funded under 
the 2nd Health Programme.  

The in-depth review shows that the extent of 
linkages between actions funded through the 
Health Programme and other EU programmes 
is limited. The review process was able to 
ascertain linkages in 19% of actions and show 
that in 35% there were none; but in nearly half 
(46%) “don’t know” was selected.  

3rd HP: 

Explicit reference to supporting complementary 
funding programmes as per thematic priorities in 
annex 1 of the Commission Decision for 3rd HP 

40 

 
 
 
 
 
 
 
ANNEX III – JOINT ACTIONS ACHIEVEMENTS 2008 – 2013 

Annex III – JOINT ACTIONS ACHIEVEMENTS 2008 – 2013JOINT ACTIONS 
ACHIEVEMENTS 2008 – 2013 

2008 

NAME OF JOINT ACTION 

ACHIEVEMENTS OF JOINT ACTION 

Joint Action to support the Rare 
Diseases Task Force Scientific 
Secretariat and contribute to the 
revision of the International 
Classification of Diseases in the field of 
rare diseases (RDTF Scientific support) 

Effective  scientific  and  technical  support  was  provided  to  the 
RDTF/EUCERD  and  an  important  contribution  to  revising  the  ICD  in 
order  to  make  rare  diseases  more  visible.  The  JA  raised  awareness 
of rare diseases and contributed to shaping national and EU policies, 
especially  with  its  three  Annual  Reports  on  the  State  of  the  Art  of 
Rare Disease Activities in Europe. 

Joint Action for European Community 
Health Indicators and Monitoring(ECHIM 
JA) 

The  main  result  is  the  2012  version  of  the  ECHI  shortlist  of  88 
indicators,  each  now  with  a  fully  updated  documentation  list  that 
defines  its  method  of  calculation  and  best  available  data  source, 
taking  into  account  methodological  quality,  availability  of  data,  the 
burden  for  Member  States  if  new  or  altered  data  collection  is 
needed, and the political importance of the indicator. The ECHIM JA 
also  compiled  information  on  the  status  of  implementation  in  most 
European countries, and analysed health indicators not published in 
international databases. 

2009 

NAME OF JOINT ACTION 

ACHIEVEMENTS OF JOINT ACTION 

Safety evaluation of manufactured 
nanomaterials by characterisation of 
their potential genotoxic hazard  
(NANOGENOTOX) 

European Health Examination  Survey 
Pilot Joint Action (EHES JA) 

European network for Health Technology 
Assessment (HTA) Joint  Action 
(EUnetHTA) 

The  main  outcome  was  a  set  of  standard  operating  procedures  for 
the  rapid  characterisation  of  different  types  of  manufactured 
nanomaterials  in  terms  of  their  potential  genotoxicity,  a  dispersion 
method  for  producing  suitable  media  for  exposure  to  these 
nanomaterials,  and  datasets  of  the  physico-chemical  properties  of 
the 
in  vitro 
genotoxicity  data  on  the  tested  nanomaterials  that  can  be  used  for 
risk  assessment  of  nanomaterials  in  a  variety  of  applications, 
including medicinal and consumer products. 

tested  nanomaterials.  The  JA  also  generated 

The  main  outcomes  from  the  EHES  JA  were  national  plans  for  the 
Health  Examination  Survey  (HES)  and  reports  of  the  national  pilot 
surveys.  Germany, Italy, Netherlands and England carried out full-
size national HES in parallel with the JA. Slovakia conducted its full-
scale  HES  in  2011,  and  Finland  in  2012.  The  Czech  Republic, 
Greece,  Malta,  Poland  and  Portugal  completed  theirs  in  2013-14. 
Two  countries  that  were  not  partners  of  the  JA  conducted  national 
surveys using the EHES procedures: Luxembourg (2012) and France 
(2014). 

The  JA  developed  a  background  review  and  an  HTA  Core  Model  for 
rapid  "Relative  Effectiveness  Assessment"  of  pharmaceuticals, 
together  with  a  report  on  national  HTA  strategies.  The  report  on 
training  needs  was  followed  by  a  training  workshop  on  EUnetHTA 
tools. The JA also collaborated with the European Medicines Agency 
(EMA) to improve the European Public Assessment Reports (EPARs), 
which are the full scientific assessment report published by EMA for 
every medicine, granted a central marketing authorisation. 

2010 

NAME OF JOINT ACTION 

ACHIEVEMENTS OF JOINT ACTION 

Mutual Organ Donation and 
Transplantation Exchanges: Improving 
and developing cadaveric organ 
donation and transplantation 

programmes (MODE) 

MODE’s main achievement was better knowledge of the quality and 
effectiveness of the systems for organ donation and transplantation 
in-depth 
in  participating  countries.  The  participants  gained 
knowledge of the training given to healthcare personnel of individual 
countries, in order to meet their specific needs. 

QUANDHIP’s main achievement was to create an effective laboratory 
network able to respond to outbreaks of highly infectious pathogens. 
This  was  achieved  by  developing  guidance  outlining  the  activation 
and  response  processes  undertaken  by  established  expertise  within 

41 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Quality Assurance Exercises and 
Networking on the Detection of Highly 
Infectious Pathogens  (QUANDHIP) 

Alzheimer’s Cooperative Valuation in 
Europe (ALCOVE) 

European Partnership for Action Against 
Cancer (EPAAC) 

Joint Action on Health Inequalities 
(Equity Action) 

European Surveillance of Congenital 
Anomalies (EUROCAT) 

Joint Action on Monitoring Injuries in 
Europe (JAMIE) 

Development of the European portal of 
rare diseases and orphan drugs 
(ORPHANET Europe) 

European Health and Life Expectancy 
Information System (EHLEIS) 

the  laboratory  network.  Through  this  guidance,  the  European 
laboratories  network  is  expected  to  manage  possible  future  natural 
and deliberate outbreaks of high-risk pathogens. The JA’s diagnostic 
tools  are  also  expected  to  boost  the  capacity  of  global  health 
security  systems,  and  to  address  the  laboratory  core  capacities 
required by the WHO International Health Regulations. 

As  a  result  of  its  investigations,  ALCOVE  found  that  the  number  of 
people  living with dementia in the EU was about 22.1%  lower than 
had  been  suggested  by  other  earlier  estimations.  Based  on  this 
work, the Joint Action identified a set of recommendations for future 
In  addition,  ALCOVE  produced  detailed 
data 
recommendations  in  other  areas,  namely:  timely  diagnosis  of 
dementia  and  interventions  to  provide  support  systems,  including 
ambulatory  ones,  for  the  management  of  the  behavioural  and 
psychological symptoms of dementia.  

collections. 

The  main  result  of  the  JA  was  the  ‘European  Guide  for  Quality 
National Cancer Control Programmes’, which provides an outline for 
policy-makers  on  the  basic  tenets  of  cancer  control  policy.  EPAAC 
also  provided  a  set  of  selected  indicators  that  enable  efficient 
monitoring of the roll-out of such programmes. The production of a 
guideline for developing a National Cancer Plan in all Member States 
was  another  major  achievement.  Results  related  to  screening  and 
early  diagnosis  included  the  creation  of  the  "European  School  for 
Screening  Management,  preparation  of  a  report  on  inequalities  in 
cancer  screening  programmes;  development  of  quality  criteria  for 
health checks and organisation of regional workshops on population-
based screening programmes. Activities under the JA also led to the 
development  of  many  European  networks  in  cancer  care.  EPAAC 
explored  ways  that  networks  can  innovatively  and  efficiently  help 
patients  at  regional,  national  and  EU  levels.  The  JA  also  designed 
the structure of a future European cancer information system (ECIS) 
and took the first step towards this (harmonisation of incidence and 
survival  data)  in  order  to  update  European  cancer  data  and 
construct  a  common  database  computing  incidence,  survival  and 
prevalence data. 

The  main outcome was  increased action and mutual  learning about 
socio-economic and area-based inequalities in health, and increased 
commitment to improving the situation in Europe. More specifically, 
the  action  led  several  countries  to  significantly  improve  their 
capacity to take action to address health inequalities, and reached a 
greater consensus on which approaches really work, plus knowledge 
and  awareness  of  tools/methods  that  help  to  promote  a  cross-
government approach to health equity. 

The EUROCAT JA supported epidemiological surveillance through the 
EUROCAT  network  of  population-based  congenital  anomaly 
registers. It was one of several JAs in the area of rare diseases, and 
aimed  to  establish  a  sustainable,  high-quality  and  easily  accessible 
information system on CA for almost one third of the European birth 
population. 

One of the JA’s main outputs was the IDB-JAMIE manual, designed 
to support national injury surveillance and reporting systems across 
the  EU.  The  manual  was  fine-tuned  through  meetings  and  the 
training of partners in applying the publication’s rules and guidelines 
in local practice. 

In  this  JA  participants  collaborated  to  make  the  "Orphanet"  portal 
the  main  reference  source  of  information  on  rare  diseases  for  all 
European  citizens.  Orphanet  now  offers  information  on  well  over  3 
000  rare  diseases,  in  English,  Dutch,  French,  German,  Italian, 
Portuguese and Spanish. 

The main outcome of the JA was a consolidated information system 
allowing  online  calculation  of  health  indicators  (prevalence,  life  and 
health  expectancies  including  healthy  life  years  (HLY),  with  health 
information  drawn  from  European  surveys.  The  partners  also 
modified  the  system’s  general  architecture  to  allow  its  future 
extension  to  national  and  sub-national  use  by  the  Member  States. 
The  JA  produced  four  series  of  country  reports  on  health 
in  the  partners’  national 
expectancy,  which  were  published 

42 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
languages.  It  also  produced  the  proceedings  of  the  EHLEIS  annual 
meetings,  to encourage Member States to  use  health expectancies, 
including  HLY,  in  their  social  policies.  The  JA  developed  new 
statistical  tools  for  attribution  and  decomposition  analyses,  and 
health  impact  assessment.  It  also  produced  technical  reports  and 
scientific analyses exploring gender variations in HLY within Europe, 
trends  over  time,  social  differentials  in  GALI  (Global  Activity 
Limitation Indicator) between Member States, and various validation 
studies of the GALI. 

JA-EHGov  successfully  contributed  to  establishing  a  dedicated 
mechanism for eHealth at EU level, thus bridging the gaps between 
governance,  strategy  and  the  operational  level  in  this  field.  The  JA 
focused  on  informing  policy  and  healthcare  decision-makers  in  the 
EU  countries  represented  in  the  eHealth  Network.  In  addition,  the 
JA-EHGov  created  an  archive  of  EU  eHealth  work  in  the  field  of 
electronic-ID, legal, semantic and technical interoperability 

Joint Action e-Health Governance 
Initiative (JA-EHGov) 

NAME OF JOINT ACTION 

ACHIEVEMENTS OF JOINT ACTION 

2011 

European Union Network for Patient 
Safety and Quality of Care (PaSQ) 

Achieving Comprehensive Coordination 
in Organ Donation 

throughout the European Union 
(ACCORD) 

EUCERD Joint Action: working for rare 
diseases (EJA) 

European Network for Health 
Technology Assessment (HTA) Joint 
Action 2 (EUnetHTA JA2) 

The  JA  has  collected  around  500  good  practices  (Patient  Safety 
Practices  and  Good  Organisational  Practices).  These  are  now 
accessible  to  healthcare  professionals  and  the  public  through  a 
mutual  learning  web  platform:  www.PaSQ.eu.  The  JA  has  also 
organised  some  35  exchange  events,  involving  experts  from  20 
Member  States.  These  events  enable  experts  and  stakeholders  to 
discuss and share selected clinical and organisational good practices 
that  they  have  selected  on  the  learning  platform.  This  led  to 
identification 
patient 
involvement/empowerment  reporting  and  learning,  rapid  exchange 
systems, quality improvement systems, and implementation of good 
clinical practices.  

the  most 

important 

priorities: 

of 

that  data  collection 

ACCORD has investigated EU countries’ experience of living donation 
and  Living  Donor  Registries  (LDRs).  It  has  also  come  up  with 
recommendations  for  LDRs  and  a  Pan-European  Registry  of  LDRs 
(data  set/dictionary  if  technical,  organisational  and  governance 
issues)  and  produced  a  web-based  platform  for  piloting  its 
recommendations  for  international  data  sharing.  The  pilot  phases 
showed 
the 
methodology  proposed  and  taking  into  account  different  types  of 
situations (file upload for countries with pre-existing register(s) and 
direct entry for countries without a register). In addition, the JA has 
produced a preliminary description of end-of-life practices applied to 
brain-injured  patients,  after  a  study  of  67  hospitals  across  15  EU 
countries;  trained  professionals  from  participating  countries  on  the 
Plan-Do-Study-Act  (PDSA)  methodology;  and  identified  areas  for 
improvement  at  participating  hospitals  (and  sometimes  at  higher 
level), whilst designing and implementing plans for improvement. 

is  possible,  according 

to 

The  key  outcome  of  the  EJA  is  a  set  of  recommendations  and 
opinions  on  critical  questions  arising  from  the  implementation  of 
policies  on  rare  disease  both  at  the  EU  and  Member  State  levels. 
Based  on  exchange  of  experience  between  Member  States’  health 
authorities,  as  well  as  from  the  contribution  of  stakeholders,  these 
recommendations  are  then  endorsed  by  the  EUCERD/Expert  Group 
and  communicated  to  national  and  European  policymakers,  patient 
organisations and professional associations. 

its 
Overall,  EUnetHTA  2  JA  has  developed  and  endorsed 
recommendations  on  a  sustainable  EU  cooperation  on  HTA.  It  has 
continued  to  influence  and  play  an  important  role  in  the  HTA 
Network, and as a result national adaptation of EUnetHTA outputs is 
gaining momentum. Partners in this JA have also further developed 
processes  for  handling  HTA  conflict  of  interest  and  confidentiality. 
Lastly, this JA organised a successful ‘HTA 2.0 Europe – Teaming Up 
for Value’ conference in Rome, Italy in October 2014, attracting 450 

43 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Cross-Border Patient Registries Initiative 
(PARENT) 

participants from all over the world. 

Besides  a  comprehensive  overview  of  the  current  situation  in  the 
EU/Member  States  regarding  patient  registries,  the  JA  has 
developed  a  coordination  mechanism  ("the  Associated  Projects 
Group")  to  exploit  synergies  between  PARENT  and  related  EU  Joint 
Actions  and  projects  on  patient  registries.  This  Group  coordinates 
work  to  ensure  that  parallel  activities  are  not  duplicating  or 
diverging  in  terms  of  methodology,  semantics,  or  policy.  It  will 
actively  encourage  decision-makers  from  associated  projects  to 
align  their  activities  and  exploit  results,  whilst  sharing  and 
exchanging  resources  with  PARENT.  In  2013,  the  JA  launched  the 
Prototype and Pilot for 

a  EU-level  relevant  source  of  information  for  national  patient 
registries,  known  as  the  PARENT  Registry  of  Registries  (parent-
RoR.eu). 

201252 

NAME OF JOINT ACTION 

ACHIEVEMENTS OF JOINT ACTION 

European Health Workforce Planning & 
Forecasting 

(EUHWforce) 

Facilitating exchange of organs donated 

in EU Member States (FOEDUS) 

The main outcome will be the consolidation of a permanent network 
for  Health  Workforce  (HWF)  planning  and  forecasting.  This  will 
support  the  EU  and  the  Member  States  as  they  work  towards  a 
stronger  and  more  effective  European  HWF.  As  a  result,  HWF 
planners  will  be  better  prepared  for  future  challenges  in  the  field, 
supported  by  a  better  prepared  educational  and  health  system. 
Further  expected  outcomes  of  this  JA  include  increased  capacity  in 
HWF  planning  and  forecasting,  data  collection  and  analysis; 
improved  data  collection,  notably  for  HWF  mobility  in  the  EU;  and 
better insight into international benchmarks on HWF. 

FOEDUS  is  expected  to  develop  an  EU-wide  common  approach  to 
organ  exchange,  plus  a  better  knowledge  of  existing  barriers 
(financial,  logistic,  and  legal)  that  are  hindering  this  practice.  By 
developing  better  practice  (e.g.  recommendations  for  international 
organ  exchange)  and  easier  exchange  of  organs  donated  in  EU 
countries, the JA is working towards increasing bilateral agreements 
among  EU  countries. This aims  to increase the number of available 
organs donated and transplanted across Europe. 

Mental Health and Well-being (MH-WB)  MH-WB  has  already  created  new  –  and  promoted  existing  – 
networks  of  relevant  stakeholders  in  participating  countries.  In 
those  countries,  it  has  analysed  the  fields  of  prevention  of 
depression  and 
community-based 
suicide,  development  of 
approaches,  mental  health  at  the  workplace,  mental  health  in 
schools,  and  integration  of  mental  health  in  all  policies.  The  JA  has 
also gathered good practices in MH-WB policy. It is now developing 
recommendations  for  action  that  will  be  widely  discussed  with  all 
relevant stakeholders in Europe. 

Improving Quality in HIV Prevention 
(Quality Action) 

The impact on maritime transport of 
health threats due to biological, 
chemical and radiobiological agents, 

The  expected  results  of  Quality  Action  are:  recognition  of  Quality 
Assurance/Quality Improvement (OA/OI) in strategic documents and 
forums; a commitment to integrate QA/QI at all levels; transferable, 
evidence-based,  pilot-tested,  practical  QA/QI  tools  and  training; 
capacity  to  use  QA/QI  at  the  programme  and  project  levels;  a 
sustainable  network  of  organisations  and 
trained  experts 
experienced  in  QA/QI;  guidance  on  effective  HIV  prevention  based 
on a Charter for Quality, and a policy kit. 

EU  SHIPSAN  ACT  has  already  produced  a  State  of  the  Art  report, 
with a literature review on infectious diseases, surveys on practices 
of  EU  authorities  on  chemical,  biological,  radiological  and  nuclear 
(CBRN)  incidents  in  maritime  transport,  hygiene  inspection  on 
fishing  and  inland  vessels,  and  training  needs  on  core  capacities  at 
points of entry. The action produced a ship inspection plan including 
competencies,  roles  and  responsibilities  and  code  of  conduct.  The 
outlines  of  inspection  were  revised  and  used  during  98  full 
inspections  and  two  re-inspections  on  board  of  87  cruise  ships  and 
11 ferries by the trained inspection teams. The inspections revealed 

52   The Joint Actions starting in 2012 and 2013 have not been finalized. Hence, no final results are available. 

44 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                      
including communicable diseases 
(SHIPSAN ACT) 

554  deficiencies,  including  90  non-compliances  with  EU  legislation 
and  414  non-compliances  with  European  Manual  for  Hygiene 
Standards  and  Communicable  Diseases  Surveillance  recommended 
standards.  Guidelines  will  be  produced 
to  allow  consistent 
preparedness  planning  in  the  EU  based  on  shared  and  common 
standards, 
Regulation 
implementation. 

International 

facilitating 

Health 

NAME OF JOINT ACTION 

ACHIEVEMENTS OF JOINT ACTION 

2013 

Assisted Reproductive Technologies and 
Haematopoietic stem cells 
Improvements for Quality and Safety 
throughout Europe (ARTHIQS) 

European Guide on Quality 
Improvement in Comprehensive Cancer 
Control (CANCON) 

Chronic Diseases and Promoting Healthy 
Ageing across 

the Life Cycle (JA-CHRODIS) 

Joint Action on Reducing Alcohol-related 
Harm 

(JA RARHA) 

Strengthening Collaborations for 
Operating 

Pharmacovigilance in Europe (SCOPE) 

ARTHIQS  is  making  good  progress  towards  all  goals:  creating 
capacity  at  national  level  in  all  28  EU  Member  States  for  assisted 
reproductive  technologies  and  on  haematopoietic  stem  cells  (HSC) 
partners  exchanged  their  HSC  donor  follow-up  procedures,  and 
discussed  topics  to  be  covered  by  the  guideline  for  Cordon-Blood-
Banks (CBB) authorisation. 

The  JA  builds  on  the  EPAAC  Joint  Action  and  its  first  aim  is  to 
improve  overall  cancer  control  through  quality-based  cancer 
screening  programmes,  better 
integration  of  cancer  care, 
community-based  care  approaches  and  by  providing  concerted 
efforts in all aspects of survivorship, including palliative care. These 
key elements will be combined with other relevant aspects of cancer 
control  to  develop  a  European  Guide  on  Quality  Improvement  in 
Comprehensive  Cancer  Control.  The  second  main  aim  is  to  discuss 
key  cancer  control  topics  through  the  Member  State  Platform, 
leading  to  position  papers  that  all  EU  countries  can  use  when 
developing their own cancer-control policies.  

Once  completed,  JA-CHRODIS  is  expected  to  deliver  a  mechanism 
for  the  collection,  validation,  scaling  up  and  transfer  of  good 
practices  on  chronic  care.  It  will  have  paid  particular  attention  to 
health  promotion  and  the  prevention  of  chronic  conditions,  multi-
morbidity and diabetes. Concrete outputs will include a platform for 
knowledge  exchange,  including  a  help  desk  and  a  clearinghouse,  a 
training  programme  for  health  professionals  to  address  multi-
morbidity,  a  set  of  best  practices  on  primary  prevention,  early 
detection,  secondary  prevention  and  management  of  diabetes, 
including  patient  empowerment  programmes  and  a  review  of 
existing national programmes on diabetes. 

The  JA  contributes  to  capacity  building  among  partners  and  in  the 
wider  public  health  community.  It  strengthens  capacity  in  alcohol 
survey  methodology  and  provides  a  common  instrument  for 
monitoring progress in reducing alcohol related harm. In addition it 
clarifies  the  scientific  basis  and  practical  implications  of  drinking 
guidelines  as  a  public  health  measure.  It  increases  consensus  on 
key  messages  about  harmful  drinking  to  the  population  and  health 
professionals.  It  also  enhances  access  to  well  described,  likely 
transferable  interventions  on  which  some  evidence  of  effectiveness 
in  influencing  attitudes  or  behaviour  and  cost  estimates  are 
available.  The  tools  developed  in  the  JA  help  plan  public  health 
policies  that  in  the  longer  term  contribute  to  reducing  alcohol 
related harm, the risk of chronic diseases and the burden for health 
systems. 

 SCOPE  is  expected  to  provide  benchmarking  of  different  EU 
systems  through  audit  and  survey  work  to  identify  best  practices 
and  weaknesses  in  the  national  pharmacovigilance  systems.  It  will 
also create a network among Member States to share and document 
best practice through fora  under different  topics. In addition,  it will 
develop tools and guidance to support best practice. These will help 
Member States to identify and manage efficient use of resources for 
operation  of  their  quality  system,  whilst  helping  them  to  plan 
capacity for future pharmacovigilance assessment work. 

45 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX IV – QUALITY ASSESSMENT FORM 

QUALITY ASSESSMENT FORM53 

The  present  quality  assessment  is  a  synthesis  of  assessments  carried 
out by the Steering Group members such as the Sec Gen, the DG SANTE 
evaluation  function,  other  DG  SANTE  units  and  external  health 
stakeholders  

Date of the Quality Assessment 21/07/2015 

(1) RELEVANCE 
Does the evaluation respond to information needs, in particular as expressed in the terms of references? 

SCORING   

Poor 

 Satisfactory 

Good 

Very Good   

Excellent           

X 

Arguments for scoring:    The evaluation corresponds to the needs as expressed in the Terms of 
Reference though a number of issues have not been fully addressed.  Limitations in scope are 
discussed and justified.    

If  relevant:  Contextual  (such  as    deficient  terms  of  references)  and  contractual  constraints  (such  as    lack  of  time, 
insufficient resources) 

53 Refer to the ‘Guide on Scoring the Criteria’ for how to assess each criterion 

46 

 
 
 
 
 
 
 
 
                                                      
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
(2) APPROPRIATE DESIGN  
Is  the  design  of  the  evaluation  adequate  for  obtaining  the  results  needed  to  answer  the  evaluation 
questions? 

SCORING   

Poor 

 Satisfactory 

Good 

Very Good   

Excellent           

X 

Arguments for scoring:    The applied methodology follows the approach set out in the Terms of 
Reference, and in some cases was more ambitious, even if at the end some of these ambitions 
were  not  totally  satisfied  (public  health  capacity  of  MS).  The  indicators  were  defined  after 
discussion with all Steering Group members.  

If relevant: Contextual (unexpected issues) and contractual constraints (such as  lack of time and resources) 

(3) RELIABLE DATA  
Are data collected adequate for their intended use and have their reliability been ascertained? 

SCORING   

Poor 

Satisfactory 

Good 

Very Good   

Excellent           

X 

Arguments  for  scoring:            There  were  a  lot  of  interviews  but  also  data  collected  via  open 
sources. Tools and data collection limitations are discussed and explained.  The combination 
of qualitative and quantitative data is adequate to strengthening the evaluation. The quality 
was controlled internally and by the Steering Group.  

If relevant: Contextual (such as  lack of data or access to data base) and contractual constraints (such as  lack of time 
and resources) 

(4) SOUND ANALYSIS  
Are data systematically analysed to answer evaluation questions and cover other information needs in a 
valid manner?  

SCORING   

Poor 

Satisfactory  

Good 

Very Good   

Excellent           

X 

47 

 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
Arguments for scoring:    The analysis uses appropriate quantitative and qualitative techniques 
suitable  to  the  evaluation  context.  Cross  checking  of  findings  has  taken  place.  The  policy 
context is well taken into account in the analysis and the report reflects an appropriate range 
of stakeholders consulted.  However, the correlation analysis is not in all aspects adequate for 
the type of data available.  

If relevant: Contextual and contractual constraints (such as  lack of resources and time) 

(5) CREDIBLE FINDINGS  
Do findings follow  logically from  and are  justified  by,  the  data/information analysis and  interpretations 
based on pre-established criteria and rational?  

SCORING   

Poor 

Satisfactory  

Good 

Very Good   

Excellent           

X 

Arguments for scoring:     The evaluation presents a correct sequence. Findings are supported 
by  evidence  originating  from  sound  analysis.  Generalisations  or  extrapolations  when  made 
are  justified.  Limitations  on  validity  are  pointed  out  and  results  of  the  analysis  reflect  an 
acceptable compromise of the perceptions of stakeholders and those described by figures and 
facts observed and estimated.  

If relevant: Contextual and contractual constraints  

(6) VALID CONCLUSIONS  
 Are conclusions non-biased and fully based on findings? 

SCORING   

Poor 

Satisfactory  

Good 

Very Good   

Excellent           

X 

Arguments for scoring:       The experience of the contractors and their previous work in earlier 
evaluations  of  the  Strategy  and  the  Health  Programme  helped  the  evaluators  direct  their 
logically 
research  and  product  valid  conclusions.  Conclusions  are  coherently  and 
substantiated by evaluation findings. They are orderly presented and related. Controversial 
issues are presented in a fair and balanced manner.  

If relevant: Contextual and contractual constraints 

48 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
(7) HELPFUL RECOMENDATIONS  
Do  areas  need  improvements  identified  in  coherence  with  the  conclusions?  Are  the  suggested  options 
realistic and impartial? 

SCORING   

Poor 

Satisfactory  

Good 

Very Good   

Excellent           

X 

Arguments  for  scoring:        Recommendations  stem  logically  from  conclusions.  They  are  quite 
clear and focus on improvement very practically and not just theoretically. A certain number 
of them reiterate earlier ones not yet fully implemented and for this reason judged ex-ante as 
difficult to put into practice.     

If relevant: Contextual and contractual constraints 

(8) CLARITY 
Is the report well structured, balanced and written in an understandable manner?  

SCORING   

Poor 

Satisfactory  

Good 

Very Good   

Excellent           

X 

Arguments for scoring:      The report is professionally written and easy to follow. It is based on 
the four main blocs of the Programme implementation which are a) the management of the 
Programme,  b)  the  dissemination  practices,  c)  the  impact  and  d)  the  synergies  with  other 
Programmes and Strategies. All evaluation questions have been answered with various level 
of  satisfaction  and  the  evaluation  aspects  of  relevance,  effectiveness,  efficiency,  Eu  added 
value, etc. have not been neglected.  

 If relevant: Contextual and contractual constraints 

49 

 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
OVERALL ASSESSMENT  

OF THE FINAL EVALUATION REPORT 

Is the overall quality of the report adequate, in particular: 

  Does the evaluation fulfil contractual conditions?    YES 

  Are the findings and conclusions of the report reliable, and are there any specific limitations to their 
validity and completeness?  YES,   the specific limitations are explicitly presented in the final 
report. and discussed with the Steering Group Members. 

 

Is the information in the report potentially useful for designing intervention, setting priorities, 
allocating resources or improving interventions?  YES 

Given the contextual and contractual constraints encountered: 

  What lessons can be learned from the evaluation process?   

It  appears  to  have  been  correct  to  have  focused  the  evaluation  on  specific  issues  of  the 
programme instead of applying the traditional questions across the board.  

Such  a  more  focused  evaluation  should  have  been  conducted  already  at  mid-term.  This 
would  have  helped  DG  SANTE  to  make  explicit  the  objectives  of  the  Programme  and 
implement  them  in  a  more  concrete  way  using  the  various  financial  mechanisms  more 
consciously. 

This  evaluation  should  at  least  be  taken  into  account  in  implementing  the  3rd  Health 
programme  and  its  subsequent  calls.    It  will  be  available  on  the  web  and  communicated 
further in order to make easy to understand and improve all the process when participating 
in a call from all European partners.  

In the meanwhile, an action plan is being elaborated and will strive to put in practice the 
options for change.  

50